- Email: [email protected]
The effect of a clinical problem-solving curriculum on medical student examination performance
THE EFFECT OF A CLINICAL PROBLEM-SOLVING CURRICULUM ON MEDICAL STUDENT EXAMINATION PERFORMANCE Eugene C. Toy, MD, Joseph B. Johns, MD, Benton Baker III, MD, Patti Jayne Ross, MD, and Larry C. Gilstrap III, MD
The objective of this study was to determine whether a clinical problem-solving curriculum during the third-year obstetrics/ gynecology clerkship would affect National Board Medical Examiners (NBME) subject test performance. During the 1999 –2000 academic year, 184 third-year medical students rotated through the obstetrics/gynecology clerkship. They were assigned to one of three clinical training sites. Thirty-six students were assigned to a community hospital, whereas the remaining 148 students were assigned to either a private university hospital or a county hospital. In July 1, 1999, the community hospital adopted a problemsolving curriculum designed to stimulate a better understanding of underlying mechanisms of disease rather than the memorization of facts. Each morning, an attending physician spent 20 minutes on interactive conferences. At the end of the clerkship, each student took the NBME subject test for obstetrics and gynecology. A test score of 80 was chosen as the honors level. Students who participated in the problem-solving curriculum scored significantly higher than did those taught by the From the Department of Obstetrics, Gynecology and Reproductive Sciences, University of Texas Houston Medical School; and CHRISTUS St. Joseph Hospital Obstetrics and Gynecology Residency Program, Houston, Texas.
traditional program, based on median NBME subject test scores, 79.0 (interquartile range, 74.0 – 82.0) versus 71.0 (interquartile range, 65.0 –76.5), P < .001, and the proportion with subject scores >80, (44.4% versus 10.1%, P < .001). A clinical problem-solving curriculum may lead to higher NBME test scores. (Prim Care Update Ob/Gyns 2002:9:135–137. © 2002 Elsevier Science Inc. All rights reserved.)
National Board Medical Examiners (NBME) subject test performance has been used as a measure of medical students’ cognitive knowledge. Lecture attendance, the order of rotations, and the United States Medical Licensing Examination (USMLE) Step 1 scores have been shown to correlate to NBME scores in obstetrics and gynecology.1–3 Many medical schools have adopted evidence-based medicine and problem-based learning (PBL) strategies to encourage medical students to develop analytical and critical thinking skills.4 – 6 Teaching programs using PBL have not been shown to consistently lead to superior clinical clerkship performance.7 Critics of PBL argue that the learning may be inefficient because numerous avenues of pursuit are common with student-led inquiry and that the coverage of the material
Volume 9, Number 4, 2002 © 2002 Elsevier Science Inc., all rights reserved. 1068-607X/02/$22.00
●
is sometimes incomplete, depending on the participants in the group.8,9 To attempt to achieve a balance between traditional lectures and PBL, facilitated clinical problem-solving conferences were introduced at one of the three clinical training sites during the thirdyear obstetrics and gynecology clerkship.
Materials and Methods The third-year clerkship at the University of Texas Houston Medical School lasts 8 weeks, divided equally between the obstetrics and gynecology services. Students are assigned to one of three clinical training sites: a community hospital, a university private hospital, and a county hospital. Before each rotation, a list of students’ names was used by the clerkship coordinator for hospital assignment. Although a computer-generated randomized procedure was not used, designation was performed without regard for the students’ desire to pursue obstetrics and gynecology and with minimal regard for student preference. All students attended the same 4 hours of lectures each Monday morning at the medical school. The remainder of their instruction occurred at their individual clinical site. During the 1999 –2000 academic year, the community hospital
PII S1068-607X(02)00104-X
135
TOY ET AL Table 1. NBME Test Performance Variable
Community Hospital (n ⴝ 36)
USMLE Step 1 scores, mean (IQR) Election to AOA honor society, n (%) NBME test score, mean (IQR) NBME test score of ⱖ80, n (%)
215 (208–229) 5 (13.9%) 79.0 (74.0–82.0) 16 (44.4%)
Other Sites (n ⴝ 148)
P Value
214 (200–229) .428 24 (16.2%) .929 71.0 (65.0–76.5) ⬍.001 15 (10.1%) ⬍.001
NBME ⫽ National Board Medical Examiners; USMLE ⫽ United States Medical Licensing Examination. AOA ⫽ Alpha Omega Alpha. Data represented as median (interquartile range), or number (%).
adopted a new teaching program, emphasizing the development of rational approaches to common and/or significant medical problems. Each morning, an attending physician spent an average of 20 minutes on interactive sessions; during these conferences, students were encouraged to analyze and explore underlying mechanisms rather than to memorize facts. Participants were challenged to give their rationale for their approaches. The attending physician questioned all answers and, at the conclusion of the conference, provided feedback regarding which of the students’ assumptions were correct. The topics included obstetrics, gynecology, and primary care. Other aspects of the curriculum were unchanged. Each student took the NBME subject test for obstetrics and gynecology at the conclusion of the clerkship, and the test scores were recorded. The procedure and scoring of the examination were in accordance with the NBME Subject Examination Program. Test scores were scaled to have a mean of 70
and a standard deviation of 8 for a group of 9,000 first-time takers who took the examination as a final clerkship exam after rotations during the 1993–1994 academic year. A test score of 80 was chosen to designate the Honors level. We used SPSS version 10.0 (SPSS, Inc., Chicago, IL) to analyze the data using the Mann-Whitney U test or the 2 test, as appropriate.
Results One hundred eighty-four students rotated through the obstetrics and gynecology clerkship. Thirty-six were assigned to the community hospital, and 148 were assigned to the other sites. As compared with those taught at other sites, students who rotated at the community hospital had similar USMLE Step 1 scores, and a similar proportion were eventually elected to the Alpha Omega Alpha honor society (Table 1). Students who participated in the problem-solving curriculum scored significantly higher
than did those who received the traditional instruction, with median NBME subject test scores of 79.0 versus 71.0, P ⬍ .001. Those who were taught under the new curriculum were more likely to have subject scores of ⱖ80, 16 (44.4%) of 36 examinees versus 15 (10.1%) of 148 examinees (P ⬍ .001). The median USMLE Step 1 scores were not significantly different across sites or from the academic year before the new curriculum (1998 –1999) versus the 1999 –2000 academic year (Table 2). Analysis (Table 3) of the two prior academic years (1997–1998 and 1998 –1999) revealed similar NBME test scores for students assigned to the community hospital as compared with those taught at other sites (1997– 1998 median NBME subject test scores 71.1 versus 71.0, P ⫽ .864; and 1998 –1999 median NBME subject test scores 73.2 versus 72.0, P ⫽ .580). Subject test scores of students instructed at other sites did not significantly change from the academic years 1997–1998, 1998 – 1999, or 1999 –2000. In contrast, the students who were instructed with the problem-solving program had significantly higher scores as compared with those who had been assigned to the community hospital during either of the two prior academic years.
Discussion Table 2. USMLE Step 1 Performance Variable 1998–1999 n USMLE score Median Interquartile range 1999–2000 n USMLE score Median Interquartile range
Community Hospital
Other Sites
P Value
36
167
.712
211 197–225
209 197–224
36
148
215 208–229
214 200–229
.428
Comparison of performance from 1998 –1999 to 1999 –2000 community hospital median scores and median scores from other sites showed no significant differences.
136
Our study demonstrated that students who were taught by the problem-solving curriculum had significantly higher NBME subject test scores. Although student assignment was not random, the USMLE Step 1 scores and the proportion of students elected to the AOA honor society were similar between both groups, which suggests that the two groups were comparable. Additionally, the significant improvement in subject test scores when compared Prim Care Update Ob/Gyns
PROBLEM-SOLVING CURRICULUM AND THE USMLE Table 3. NBME Test Performance Variable 1997–1998 n NBME test score Median Interquartile range 1998–1999 n NBME test score Median Interquartile range 1999–2000 n NBME test score Median Interquartile range
Community Hospital 36
Other Sites
P Value
154
71.1* 63.5–75.0 36
36
.580
⬍.001
71.0†§ 65.0–76.5
* Comparison of community hospital scores from 1997–1998 and from 1999 –2000, P ⫽ .003. ‡ Comparison of community hospital scores from 1998 –1999 and from 1999 –2000, P ⫽ .008. † Comparison of scores from other sites from 1997–1998 and from 1999 –2000, P ⫽ .890. § Comparison of scores from other sites from 1998 –1999 and from 1999 –2000, P ⫽ .831.
with the community hospital NBME test scores from the preceding two academic years indicates an influence other than an inherent site-specific learning advantage. In developing our curriculum, we sought to cover a breadth of material as well as to encourage critical thinking skills and the ability to ask discriminating questions. An emphasis was placed on the students’ understanding of the correct reasoning and pathophysiology. The participants were asked for their reasons rather than simply for the correct answers. Immediate feedback was given and direction was provided for reading assignments. Students regularly remarked that they remembered the facts much better by reviewing the mechanisms of disease processes. Also, they enjoyed having the conferences in the morning, so that they could reflect on the clinical problems throughout the day. Likewise, participants commented that they would discuss their clinical approaches with other residents and faculty. In contrast to reports that explanations are not important in problem solving, we
Volume 9, Number 4, 2002
3.
72.0§ 66.0–77.0 148
79.0*‡ 74.0–82.0
.864
71.0† 66.3–78.0 167
73.2‡ 67.0–77.5
2.
found that students often would guess at the right answer but have a flawed approach.10 PBL has been noted to be an outstanding model to self-learning, but incorporating its methodology for an entire obstetrics/gynecology curriculum is time-consuming.11 Our teaching program attempts to emulate the efficiency of traditional teaching in covering a breadth of material, yet emphasizing an understanding of underlying mechanisms of disease. Although this study did not assign students using a computergenerated list, the student designations appeared to be equitable. Furthermore, although the objective multiple choice test scores improved with this curriculum, we did not evaluate problem-solving nor clinical skills. Future studies with larger numbers of participants are needed to further assess this teaching tool. References 1. Riggs JW, Blanco JD. Is there a relation between student lecture attendance and clinical science subject
4.
5.
6.
7.
8.
9.
10.
11.
examination score? Obstet Gynecol 1994;84:311–3. Armstrong A, Dahl C, Haffner W. Predictors of performance on the National Board Medical Examiners obstetrics and gynecology subject examination. Obstet Gynecol 1998; 91:1021–2. Manetta A, Manetta E, Enman D, Keegan KA, Williams JH. Effects of rotation discipline on medical student grades in obstetrics and gynecology throughout the academic year. Obstet Gynecol 1993;169: 1215–7. Grimes DA. Introducing evidencebased medicine into a department of obstetrics and gynecology. Obstet Gynecol 1995;86:451–7. Robertson PA, Brown JS, Flanagan TA, et al. The women’s health curriculum by a problem-based learning method for medical students at the University of California, San Francisco. Am J Obstet Gynecol 1997;176:1368 –73. Scott JN, Markert RJ, Dunn MM. Critical thinking: change during medical school and relationship to performance in clinical clerkships. Med Educ 1998;32:14 – 8. Phelan ST, Jackson JR, Berner ES. Comparison of problem-based and traditional education on student performance in the obstetrics and gynecology clerkship. Obstet Gynecol 1993;82:159 – 61. Hak T, Maguire P. Group process: the black box of studies on problembased learning. Acad Med 2000;75: 769 –72. Mpofu DJ, Das M, Murdoch JC, Lanphear JH. Effectiveness of problems used in problem-based learning. Med Educ 1997;31:330 – 4. Schwartz PL, Loten EG. Brief problem-solving questions in medical school examinations: is it necessary for students to explain their answers? Med Educ 1999;33:823–7. Dwinnel BG, Damas L. Problembased learning in medical education. Hosp Pract (Off Ed) 1998;33: 15– 6, 21.
Address correspondence and reprint requests to Eugene C. Toy, MD, CHRISTUS St. Joseph Hospital, 1819 Crawford Street, Suite 1708, Houston, TX 77002.
137
The objective of this study was to determine whether a clinical problem-solving curriculum during the third-year obstetrics/ gynecology clerkship would affect National Board Medical Examiners (NBME) subject test performance. During the 1999 –2000 academic year, 184 third-year medical students rotated through the obstetrics/gynecology clerkship. They were assigned to one of three clinical training sites. Thirty-six students were assigned to a community hospital, whereas the remaining 148 students were assigned to either a private university hospital or a county hospital. In July 1, 1999, the community hospital adopted a problemsolving curriculum designed to stimulate a better understanding of underlying mechanisms of disease rather than the memorization of facts. Each morning, an attending physician spent 20 minutes on interactive conferences. At the end of the clerkship, each student took the NBME subject test for obstetrics and gynecology. A test score of 80 was chosen as the honors level. Students who participated in the problem-solving curriculum scored significantly higher than did those taught by the From the Department of Obstetrics, Gynecology and Reproductive Sciences, University of Texas Houston Medical School; and CHRISTUS St. Joseph Hospital Obstetrics and Gynecology Residency Program, Houston, Texas.
traditional program, based on median NBME subject test scores, 79.0 (interquartile range, 74.0 – 82.0) versus 71.0 (interquartile range, 65.0 –76.5), P < .001, and the proportion with subject scores >80, (44.4% versus 10.1%, P < .001). A clinical problem-solving curriculum may lead to higher NBME test scores. (Prim Care Update Ob/Gyns 2002:9:135–137. © 2002 Elsevier Science Inc. All rights reserved.)
National Board Medical Examiners (NBME) subject test performance has been used as a measure of medical students’ cognitive knowledge. Lecture attendance, the order of rotations, and the United States Medical Licensing Examination (USMLE) Step 1 scores have been shown to correlate to NBME scores in obstetrics and gynecology.1–3 Many medical schools have adopted evidence-based medicine and problem-based learning (PBL) strategies to encourage medical students to develop analytical and critical thinking skills.4 – 6 Teaching programs using PBL have not been shown to consistently lead to superior clinical clerkship performance.7 Critics of PBL argue that the learning may be inefficient because numerous avenues of pursuit are common with student-led inquiry and that the coverage of the material
Volume 9, Number 4, 2002 © 2002 Elsevier Science Inc., all rights reserved. 1068-607X/02/$22.00
●
is sometimes incomplete, depending on the participants in the group.8,9 To attempt to achieve a balance between traditional lectures and PBL, facilitated clinical problem-solving conferences were introduced at one of the three clinical training sites during the thirdyear obstetrics and gynecology clerkship.
Materials and Methods The third-year clerkship at the University of Texas Houston Medical School lasts 8 weeks, divided equally between the obstetrics and gynecology services. Students are assigned to one of three clinical training sites: a community hospital, a university private hospital, and a county hospital. Before each rotation, a list of students’ names was used by the clerkship coordinator for hospital assignment. Although a computer-generated randomized procedure was not used, designation was performed without regard for the students’ desire to pursue obstetrics and gynecology and with minimal regard for student preference. All students attended the same 4 hours of lectures each Monday morning at the medical school. The remainder of their instruction occurred at their individual clinical site. During the 1999 –2000 academic year, the community hospital
PII S1068-607X(02)00104-X
135
TOY ET AL Table 1. NBME Test Performance Variable
Community Hospital (n ⴝ 36)
USMLE Step 1 scores, mean (IQR) Election to AOA honor society, n (%) NBME test score, mean (IQR) NBME test score of ⱖ80, n (%)
215 (208–229) 5 (13.9%) 79.0 (74.0–82.0) 16 (44.4%)
Other Sites (n ⴝ 148)
P Value
214 (200–229) .428 24 (16.2%) .929 71.0 (65.0–76.5) ⬍.001 15 (10.1%) ⬍.001
NBME ⫽ National Board Medical Examiners; USMLE ⫽ United States Medical Licensing Examination. AOA ⫽ Alpha Omega Alpha. Data represented as median (interquartile range), or number (%).
adopted a new teaching program, emphasizing the development of rational approaches to common and/or significant medical problems. Each morning, an attending physician spent an average of 20 minutes on interactive sessions; during these conferences, students were encouraged to analyze and explore underlying mechanisms rather than to memorize facts. Participants were challenged to give their rationale for their approaches. The attending physician questioned all answers and, at the conclusion of the conference, provided feedback regarding which of the students’ assumptions were correct. The topics included obstetrics, gynecology, and primary care. Other aspects of the curriculum were unchanged. Each student took the NBME subject test for obstetrics and gynecology at the conclusion of the clerkship, and the test scores were recorded. The procedure and scoring of the examination were in accordance with the NBME Subject Examination Program. Test scores were scaled to have a mean of 70
and a standard deviation of 8 for a group of 9,000 first-time takers who took the examination as a final clerkship exam after rotations during the 1993–1994 academic year. A test score of 80 was chosen to designate the Honors level. We used SPSS version 10.0 (SPSS, Inc., Chicago, IL) to analyze the data using the Mann-Whitney U test or the 2 test, as appropriate.
Results One hundred eighty-four students rotated through the obstetrics and gynecology clerkship. Thirty-six were assigned to the community hospital, and 148 were assigned to the other sites. As compared with those taught at other sites, students who rotated at the community hospital had similar USMLE Step 1 scores, and a similar proportion were eventually elected to the Alpha Omega Alpha honor society (Table 1). Students who participated in the problem-solving curriculum scored significantly higher
than did those who received the traditional instruction, with median NBME subject test scores of 79.0 versus 71.0, P ⬍ .001. Those who were taught under the new curriculum were more likely to have subject scores of ⱖ80, 16 (44.4%) of 36 examinees versus 15 (10.1%) of 148 examinees (P ⬍ .001). The median USMLE Step 1 scores were not significantly different across sites or from the academic year before the new curriculum (1998 –1999) versus the 1999 –2000 academic year (Table 2). Analysis (Table 3) of the two prior academic years (1997–1998 and 1998 –1999) revealed similar NBME test scores for students assigned to the community hospital as compared with those taught at other sites (1997– 1998 median NBME subject test scores 71.1 versus 71.0, P ⫽ .864; and 1998 –1999 median NBME subject test scores 73.2 versus 72.0, P ⫽ .580). Subject test scores of students instructed at other sites did not significantly change from the academic years 1997–1998, 1998 – 1999, or 1999 –2000. In contrast, the students who were instructed with the problem-solving program had significantly higher scores as compared with those who had been assigned to the community hospital during either of the two prior academic years.
Discussion Table 2. USMLE Step 1 Performance Variable 1998–1999 n USMLE score Median Interquartile range 1999–2000 n USMLE score Median Interquartile range
Community Hospital
Other Sites
P Value
36
167
.712
211 197–225
209 197–224
36
148
215 208–229
214 200–229
.428
Comparison of performance from 1998 –1999 to 1999 –2000 community hospital median scores and median scores from other sites showed no significant differences.
136
Our study demonstrated that students who were taught by the problem-solving curriculum had significantly higher NBME subject test scores. Although student assignment was not random, the USMLE Step 1 scores and the proportion of students elected to the AOA honor society were similar between both groups, which suggests that the two groups were comparable. Additionally, the significant improvement in subject test scores when compared Prim Care Update Ob/Gyns
PROBLEM-SOLVING CURRICULUM AND THE USMLE Table 3. NBME Test Performance Variable 1997–1998 n NBME test score Median Interquartile range 1998–1999 n NBME test score Median Interquartile range 1999–2000 n NBME test score Median Interquartile range
Community Hospital 36
Other Sites
P Value
154
71.1* 63.5–75.0 36
36
.580
⬍.001
71.0†§ 65.0–76.5
* Comparison of community hospital scores from 1997–1998 and from 1999 –2000, P ⫽ .003. ‡ Comparison of community hospital scores from 1998 –1999 and from 1999 –2000, P ⫽ .008. † Comparison of scores from other sites from 1997–1998 and from 1999 –2000, P ⫽ .890. § Comparison of scores from other sites from 1998 –1999 and from 1999 –2000, P ⫽ .831.
with the community hospital NBME test scores from the preceding two academic years indicates an influence other than an inherent site-specific learning advantage. In developing our curriculum, we sought to cover a breadth of material as well as to encourage critical thinking skills and the ability to ask discriminating questions. An emphasis was placed on the students’ understanding of the correct reasoning and pathophysiology. The participants were asked for their reasons rather than simply for the correct answers. Immediate feedback was given and direction was provided for reading assignments. Students regularly remarked that they remembered the facts much better by reviewing the mechanisms of disease processes. Also, they enjoyed having the conferences in the morning, so that they could reflect on the clinical problems throughout the day. Likewise, participants commented that they would discuss their clinical approaches with other residents and faculty. In contrast to reports that explanations are not important in problem solving, we
Volume 9, Number 4, 2002
3.
72.0§ 66.0–77.0 148
79.0*‡ 74.0–82.0
.864
71.0† 66.3–78.0 167
73.2‡ 67.0–77.5
2.
found that students often would guess at the right answer but have a flawed approach.10 PBL has been noted to be an outstanding model to self-learning, but incorporating its methodology for an entire obstetrics/gynecology curriculum is time-consuming.11 Our teaching program attempts to emulate the efficiency of traditional teaching in covering a breadth of material, yet emphasizing an understanding of underlying mechanisms of disease. Although this study did not assign students using a computergenerated list, the student designations appeared to be equitable. Furthermore, although the objective multiple choice test scores improved with this curriculum, we did not evaluate problem-solving nor clinical skills. Future studies with larger numbers of participants are needed to further assess this teaching tool. References 1. Riggs JW, Blanco JD. Is there a relation between student lecture attendance and clinical science subject
4.
5.
6.
7.
8.
9.
10.
11.
examination score? Obstet Gynecol 1994;84:311–3. Armstrong A, Dahl C, Haffner W. Predictors of performance on the National Board Medical Examiners obstetrics and gynecology subject examination. Obstet Gynecol 1998; 91:1021–2. Manetta A, Manetta E, Enman D, Keegan KA, Williams JH. Effects of rotation discipline on medical student grades in obstetrics and gynecology throughout the academic year. Obstet Gynecol 1993;169: 1215–7. Grimes DA. Introducing evidencebased medicine into a department of obstetrics and gynecology. Obstet Gynecol 1995;86:451–7. Robertson PA, Brown JS, Flanagan TA, et al. The women’s health curriculum by a problem-based learning method for medical students at the University of California, San Francisco. Am J Obstet Gynecol 1997;176:1368 –73. Scott JN, Markert RJ, Dunn MM. Critical thinking: change during medical school and relationship to performance in clinical clerkships. Med Educ 1998;32:14 – 8. Phelan ST, Jackson JR, Berner ES. Comparison of problem-based and traditional education on student performance in the obstetrics and gynecology clerkship. Obstet Gynecol 1993;82:159 – 61. Hak T, Maguire P. Group process: the black box of studies on problembased learning. Acad Med 2000;75: 769 –72. Mpofu DJ, Das M, Murdoch JC, Lanphear JH. Effectiveness of problems used in problem-based learning. Med Educ 1997;31:330 – 4. Schwartz PL, Loten EG. Brief problem-solving questions in medical school examinations: is it necessary for students to explain their answers? Med Educ 1999;33:823–7. Dwinnel BG, Damas L. Problembased learning in medical education. Hosp Pract (Off Ed) 1998;33: 15– 6, 21.
Address correspondence and reprint requests to Eugene C. Toy, MD, CHRISTUS St. Joseph Hospital, 1819 Crawford Street, Suite 1708, Houston, TX 77002.
137