- Email: [email protected]
The Effect of the Duration and Structure of a Surgery Clerkship on Student Performance
Journal of Surgical Research 84, 106 –111 (1999) Article ID jsre.1999.5624, available online at http://www.idealibrary.com on
The Effect of the Duration and Structure of a Surgery Clerkship on Student Performance D. Scott Lind, M.D., 1 Tiffany Marum, M.S., Daniel Ledbetter, M.D., Timothy C. Flynn, M.D., Lynn J. Romrell, Ph.D., and Edward M. Copeland III, M.D. Department of Surgery, University of Florida, Gainesville, Florida 32610 Presented at the Annual Meeting of the Association for Academic Surgery, Seattle, Washington, November 18 –22, 1998
The emphasis on a generalist professional education has led to shortening and restructuring of the surgery clerkship in the curricula of many medical schools. Little data exist regarding the effect of these changes on student performance. Therefore, we examined the effect of the length, timing, and content of the third year surgery rotation on several clerkship and postclerkship performance measures of 487 students from July 1994 to July 1998. In addition, students’ perceptions regarding their ability to understand surgical disease topics were surveyed. The 8-week clerkship (n 5 232) was associated with higher NMBE surgery test scores (510.5 6 6.3 versus 457.4 6 10.0, P < 0.05) resulting in higher final clerkship grades (5.15 6 0.04 versus 4.87 6 0.03, P < 0.05). Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores, the longer clerkship was associated with higher total (70.6 6 0.37 versus 68.8 6 0.50, P < 0.05) and abdominal pain station (81.87 6 0.71 versus 79.54 6 0.73, P < 0.05) OCSE scores. Students rotating on surgery during the second half of third year (n 5 233) had higher NMBE surgery test scores (513.1 6 8.9 versus 460.5 6 11.2, P < 0.05) and final grades (5.17 6 0.03 versus 4.81 6 0.04, P < 0.05). Although the timing of the surgery clerkship did not significantly affect total OSCE scores, students who rotated on surgery in the second half of third year performed significantly better year on the abdominal pain OSCE station (80.47 6 0.92 versus 76.49 6 1.27, P < 0.05). Students who rotated on general surgery (n 5 298) performed significantly better on the NBME surgery test (525.6 6 6.0 versus 459.6 6 9.1, P < 0.05), although this did not significantly affect the final grade. Although general versus subspecialty surgery 1 To whom correspondence should be addressed at P.O. Box 100286, Department of Surgery, University of Florida, Gainesville, FL 32610. Fax: (352) 338-9809.
0022-4804/99 $30.00 Copyright © 1999 by Academic Press All rights of reproduction in any form reserved.
rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station (81.21 6 0.91 versus 78.17 6 0.32, P < 0.05). The length, timing, and content of the third year surgery rotation had no significant effect on performance on the oral examination. Students who had a 6-week clerkship and students who lacked exposure to general surgery felt their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery. The length, timing, and content of the surgery clerkship affect some clerkship performance measures and student perceptions of their understanding of surgical disease topics. While cognitive differences related to clerkship length are no longer detectable at the end of the third year of medical school, differences related to the acquisition of some clinical skills persist after the surgery clerkship. © 1999 Academic Press Key Words: clerkship; NMBE surgery test; USMLE step 2, OSCE. INTRODUCTION
The emphasis on a generalist professional education has led to shortening the third year surgery clerkship in the curricula of many medical schools (Fig. 1). Studies examining the effect of clerkship characteristics, such as clerkship length, on student performance have produced conflicting results [1– 4]. The differences between these studies may be due to variations in (1) the classification of clerkship variables (i.e., 12 weeks versus 8 weeks), (2) the indicators of educational outcome, and (3) school-specific differences with respect to students, curricula, and educational methods. Changes in the curriculum at the University of Florida prompted an analysis of the effect of the length, timing, and
106
107
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
TABLE 1 The Curricular Changes in the Third Year at the University of Florida Length (weeks)
FIG. 1. The change in surgery and primary care (family practice, internal medicine, pediatrics, and obstetrics/gynecology) clerkship length in U.S. medical schools. Source: The American Association of Medical Colleges (AAMC) Clerkship Directory.
content of the third year surgery rotation on several student performance measures over a 4-year period that spanned the recent curricular changes. METHODS The performance of 487 students from July 1994 to July 1998 was assessed. In 1996 the curriculum at the University of Florida underwent significant changes that resulted in a shortening of the third year surgery clerkship from 8 to 6 weeks (Table 1). Prior to this curtailment, the 8-week surgery rotation consisted of two 4-week rotations on the surgical services listed in Table 2 (1994 –1996). In July of 1996 students began rotating for 6 weeks on the surgical services listed in Table 2 (1996 –1998). Clerkship performance measures examined were the National Board of Medical Examiners (NBME) surgery test (25% of the final grade), an oral examination (25% of the final grade), and the surgery final grade. The ward evaluation of student performance was not part of this analysis but it composes 50% of the final grade. The NBME surgery test is a 2-h 120-item multiple-choice examination administered on the last day of the clerkship. NBME total scores have been shown to have a reliability of 0.80 [5]. The format of the oral examination consists of a faculty member questioning the student regarding the evaluation and management of two topics from a list of common surgical problems (30 min per topic). Ward evaluations are made by faculty utilizing a clinical evaluation form consisting of 10 items considered to be important components of professional competence (i.e., fund of knowledge, problem-solving, clinical judgement, etc.). Postclerkship performance measures included the United States Medical Licensing Examination (USMLE) step 2 and an Objective Structured Clinical Examination (OSCE). The OSCE is completed by students at the end of third year to provide a performance-based assessment of knowledge and skills acquired during the third year clinical rotations. The OSCE consists of 9 to 12 stations (abdominal pain, back pain, and shortness of breath, etc.) involving standardized patients who evaluate student performance using a checklist format. In addition, at the end of the third year, the students are required to take the USMLE step 2. The USMLE step 2 is a 2-day multiplechoice question (MCQ) examination that stresses integration of information across the medical disciplines. USMLE step 2 has been shown to have a reliability of 0.97 [5]. The manner in which USMLE step 2 scores are reported to the institution did not permit an analysis of the effect of the timing and content of the surgery clerkship on this performance measure. Only the first examination scores were
Clerkship
1994–1996
Medicine Anesthesiology Geriatrics Neurology ACLS Primary Care Comm. Medicine Pediatrics Ob-Gyn Psychiatry Surgery
8 1
1996–1998 M A G N A
2
12
6
12
8 8 7 8 44
6 6 6 6 44
Note. MAGNA, medicine, anesthesiology, geriatrics, neurology, and advanced cardiac life support (ACLS).
used for analysis for students who required retesting for any performance parameter. To examine of the effect of clerkship timing on student performance, students were divided into those students who rotated on surgery during the first half of the third year of medical school (July to January) versus those students who rotated on surgery during the last half of the academic year (January to July). Clerkship content was categorized as general surgery if students rotated on general surgery for the entire 6-week rotation or for 4 weeks as part of the 8-week surgery rotation (i.e., 4 weeks subspecialty service and 4 weeks general surgery service). Students who lacked any exposure to a general surgery service were categorized as subspecialty rotation students. Surgery rotations (i.e., subspecialty versus general) were determined by a lottery system. Data was analyzed by the Student t test and a P value of 0.05 was used for significance.
TABLE 2 Structure of the Surgery Clerkship Surgery service
1994–1996 a
1996–1998 b
General surgery 1 General surgery 6 General surgery 8 VA general surgery General surgery (JAX) Vascular VA vascular Vascular (JAX) Plastics Plastics (JAX) Transplant Urology Urology (JAX) Pediatric surgery
X X
X X X X X X X
a b
Two 4-week rotations. One 6-week rotation.
X X X X X X X X X X X
X X X X
108
JOURNAL OF SURGICAL RESEARCH: VOL. 84, NO. 1, JUNE 1, 1999
FIG. 2. (A) The effect of clerkship length on clerkship performance measures. Data represent mean 6 SEM. *P , 0.05. (B) The effect of clerkship length on USMLE step 2 total score and surgery subsection score. (C) The effects of clerkship length on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
RESULTS
Clerkship Length and Student Performance Figure 2A demonstrates that the 8-weeks clerkship (n 5 232) produced higher NMBE surgery test scores (510.5 6 6.3 versus 457.4 6 10.0, P , 0.05) resulting in higher final clerkship grades (5.15 6 0.04 versus 4.87 6 0.03, P , 0.05), but it had no significant effect on oral examination scores. Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores (Fig. 2B), the longer clerkship was associated with higher total (70.6 6 0.37 versus 68.8 6 0.50, P , 0.05) and abdominal pain station (81.87 6 0.71 versus 79.54 6 0.73, P , 0.05) OCSE scores assessed at the end of the third year of medical school (Fig. 2C).
Clerkship Timing and Student Performance Figure 3A shows that students rotating on surgery during the second half of third year (n 5 233) had higher NMBE surgery test scores (513.1 6 8.9 versus 460.5 6 11.2, P , 0.05) and final grades (5.17 6 0.03 versus 4.81 6 .04, P , 0.05), but the timing of the surgery clerkship had no significant effect on oral examination scores. Although the timing of the surgery clerkship did not significantly affect total OSCE scores, students who rotated on surgery in the second half of third year performed significantly better on the abdominal pain OSCE station (80.47 6 0.92 versus 76.49 6 1.27, P , 0.05) than students who rotated on surgery in the first half of the third year (Fig. 3B).
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
109
FIG. 3. (A) The effect of clerkship timing on clerkship performance measures. *P , 0.05. (B) The effects of clerkship timing on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
Clerkship Content and Student Performance Students who rotated on general surgery (n 5 298) performed significantly better on the NBME surgery test (525.6 6 6.0 versus 459.6 6 9.1, P , 0.05), although general versus subspecialty rotation did not significantly affect performance on the oral surgery examination or the final grade (Fig. 4A). Although general versus subspecialty surgery rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station (81.21 6 0.91 versus 78.17 6 0.32, P , 0.05) (Fig. 4B). Survey Analysis Response to the student survey was 68% and there was no difference in the response rate with respect to clerkship length or content (general versus subspecialty). Students who had a 6-week clerkship or students who lacked exposure to general surgery felt that their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery (Table 3). DISCUSSION
The national agenda to increase the number of generalist physicians has led to shortening the surgery clerkship in the curricula of many medical schools (Fig. 1). Studies examining the effect of clerkship curtailment on student performance have produced conflicting results. Smith et al. found that shortening the obstetrics and gynecology clerkship from 8 weeks to 6 weeks had no effect on the NMBE obstetrics and gynecology subject test performance [1]. On the other hand,
investigators from the National Board of Medical Examiners found that students from 32 schools with a 12-week surgery clerkship performed significantly better on the NBME surgery test and USMLE step 2 than students from 25 schools with an 8-week surgery clerkship [5]. This study failed, however, to address whether clerkship curtailment from 8 weeks to 6 weeks produced similar differences in student performance and this study only assessed cognitive measures of clinical competency. In addition, data from a national study may have limited applicability to individual medical schools because of school-specific curricular factors. Furthermore, little data exist regarding the relationship between clerkship variables and postclerkship measures of student performance. In our study, the shortening of the surgery clerkship from 8 weeks to 6 weeks significantly affected student performance on the NBME surgery test. Since clerkship curtailment had no effect on performance on the oral examination or ward (faculty) evaluations (data not shown), the effect of clerkship duration on final grade was solely the result of the lower NBME surgery test scores. Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores, the longer clerkship was associated with higher total and abdominal pain station OSCE scores assessed at the end of the third year of medical school. Thus, it appears that cognitive differences associated with the change in clerkship length are no longer detectable at the end of the third year. Several possible explanations may account for this finding. It is conceivable that differences in NBME surgery test scores associated with shortening the surgery clerkship by 2 weeks were simply attributable to the 8-week students benefiting from the extra 2 weeks of preparatory study time. Alternatively, the differ-
110
JOURNAL OF SURGICAL RESEARCH: VOL. 84, NO. 1, JUNE 1, 1999
FIG. 4. (A) The effect of clerkship content (general surgery versus subspecialty rotation) clerkship performance measures. *P , 0.05. (B) The effects of clerkship content (general surgery versus subspecialty rotation) on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
ences in NBME surgery test scores maybe related to actual learning deficiencies associated with a shortened clerkship. The inability to detect an effect of clerkship length on postclerkship USMLE step 2 total or surgery subsection scores suggests that with time students overcome the detrimental effects of the shortened clerkship on cognitive learning. Poor student performance on the NBME surgery test may have influenced students to prepare more for the USMLE step 2 examination. The longer clerkship was, however, associated with higher total and abdominal pain station OSCE scores asTABLE 3 Response to Student Survey Mean student response a Surgical topics
8 weeks
Fluid and electrolytes Wound healing Trauma Breast Acute abdomen Gastrointestinal hemorrhage Common vascular problems Perioperative care
2.8 6 1.0
a
6 weeks
General
Specialty
b
2.8 6 1.1
2.6 6 1.1*
3.0 6 1.0
1.5 6 0.9* 1.7 6 0.9* 2.0 6 1.3* 2.2 6 1.0 2.0 6 0.9*
2.2 6 1.0 2.6 6 1.0 2.7 6 1.2 2.5 6 1.2 2.6 6 1.2
1.7 6 1.0* 1.8 6 0.9* 2.5 6 1.2* 2.1 6 1.0* 2.2 6 1.0*
2.3 6 1.2 2.5 6 1.0 2.9 6 1.1 2.9 6 1.2 2.6 6 1.3
2.6 6 1.3
2.6 6 1.1
2.8 6 1.5
2.4 6 1.0
2.1 6 0.9*
2.5 6 1.1
2.1 6 0.9*
2.5 6 1.1
Using the Likert-type scale below, please respond to the statement: “I feel my surgery rotation adequately prepared me to understand the following surgical topics. . .” Strongly Agree 1 2 3 4 5 Strongly Disagree. b Data 5 mean 6 SEM. * P , 0.05 by t test.
sessed at the end of the third year of medical school, suggesting that clinical skills deficits exist postclerkship. The lack of an OSCE immediately following the surgery clerkship makes it impossible to determine precisely whether these clinical skill deficits are persistent or develop de novo postclerkship. Our study demonstrates that students rotating on surgery during the second half of third year had higher NMBE surgery test scores resulting in higher final grades. Other studies have also demonstrated that clerkship timing affects educational outcome [2, 3]. Our findings probably reflect knowledge gained by the student on previous clerkships, particularly the internal medicine clerkship, which can be applied to the NBME surgery test. This finding may also provide justification for “handicapping” students depending upon when they rotate on surgery during the third year of medical school. This handicapping may have particular importance for those students competing for a surgical residency. It is important to note, however, that the order of clerkship rotation was determined by lottery, minimizing the bias of student selection to this difference (i.e., students with surgical career interests selecting surgery at a particular time to maximize their academic performance). In this study, students who rotated on general surgery performed significantly better on the NBME surgery test but the type of rotation did not significantly affect the final grade. Although general versus subspecialty surgery rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station. The effect of the clinical setting of the surgery rotation on student performance has also been previously studied [6]. Poenaru et al. found no differ-
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
ence between students rotating on a general surgical and subspecialty service in several educational parameters. They concluded that effective undergraduate education could occur in a subspecialty setting as long as the educational process is deliberate, structured, and objective-based. The effect of a lack of a general surgery rotation on performance on the abdominal pain OSCE maybe due to a lack of exposure to the common general surgical problem in the subspecialty setting. The fact that the length, timing, and content of the surgery clerkship did not influence performance on the surgery oral examination is not surprising. Other studies have found little correlation between oral examinations and other performance measures [7]. In general, oral examinations may be unreliable instruments to discriminate between students [8]. Interestingly, our survey data complement our educational performance data. Students who had a 6-week clerkship or students who lacked exposure to general surgery felt their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery (Table 3). There are some limitations to this study. Although there were no obvious differences in instructional methods between the study groups, other factors such as differences in the use of ambulatory setting for teaching or differences in student-faculty contact between the groups may account for some of the differences detected in student performance. Alternatively, the differences in student performance may be due to educational differences that existed prior to the surgery clerkship. Analysis of student performance prior to the surgery clerkship (i.e., Medical Colleges Aptitude Test (MCAT) scores, Grade Point Averages, and USMLE step 1 scores) shows no difference between the study groups. In fact, USMLE step 1 scores at the University of Florida have steadily increased in the past 5 years [9]. In addition, our study ignores nonsurgical curricular changes that have occurred at the University of Florida (Table 1). The shortening of the pediatrics and obstetrics and gynecology clerkships may impact student performance measures at the end of the third year that assess clinical competence across many
111
medical disciplines (i.e., total OSCE score and total USMLE step 2 scores). Finally, caution should also be used before extrapolating our data to other institutions. In summary, the length, timing, and content of the surgery clerkship affect some clerkship performance measures and student perceptions of their understanding of surgical disease topics. While cognitive differences related to clerkship length are no longer detectable at the end of the third year of medical school, differences related to the acquisition of some clinical skills persist after the surgery clerkship. Data on the postgraduate educational consequences of shortening the surgery clerkship are sorely required. REFERENCES 1.
2.
3.
4.
5.
6.
7.
8.
9.
Smith, E. R., Dinh, T. V., and Anderson, G. A decrease from 8 to 6 weeks in obstetrics and gynecology clerkship: Timing on medical student’s cognitive knowledge. Obstet. Gynecol. 86: 458, 1995. Whalen, J. P., and Moses, V. K. The effect on grades of the timing and site of third year internal medicine clerkships. Acad. Med. 65: 708, 1990. Baciewicz, F. A., Arent, L., Weaver, M., Yeasting, R., and Thomford, N. R. Influence of clerkship structure and timing on individual student performance. Am. J. Surg. 159: 265. Gary, N. E., and Rosevar, G. C. Effect of reduction in length of the third clerkships on students’ academic performance. J. Med. Educ. 63: 406, 1988. Ripkey, D. R., Case, S., and Swanson, D. B. Predicting performances on the NBME surgery subject test and USMLE step 2: The effects of surgery clerkship timing length. Acad. Med. 72(10): S31, 1997. Poenaru, D., Davidson, L., Donnely, M., and Tekain, A. Is a mandatory general surgery rotation necessary in the surgical clerkship? Am. J. Surg. 175: 515, 1998. Amiel, G. E., Tann, M., Krausz, M. M., Bitterman, A., and Cohen, R. Increasing examiner involvement in an objective structured clinical examination by integrating a structured oral exam. Am. J. Surg. 173: 546, 1997. Stillman, R. M., Lane, K. M., Beeth, S., and Jaffee, B. M. Evaluation of the student. Improving validity of the oral examination. Surgery 93: 439, 1983. Watson, R. T., Suter, E., Romrell, L. J., Harmon, E. M., Rooks, L., and Neims, A. H. Moving a graveyard: How one school prepared the way for continuous curriculum renewal. Acad. Med. 73: 948, 1998.
The Effect of the Duration and Structure of a Surgery Clerkship on Student Performance D. Scott Lind, M.D., 1 Tiffany Marum, M.S., Daniel Ledbetter, M.D., Timothy C. Flynn, M.D., Lynn J. Romrell, Ph.D., and Edward M. Copeland III, M.D. Department of Surgery, University of Florida, Gainesville, Florida 32610 Presented at the Annual Meeting of the Association for Academic Surgery, Seattle, Washington, November 18 –22, 1998
The emphasis on a generalist professional education has led to shortening and restructuring of the surgery clerkship in the curricula of many medical schools. Little data exist regarding the effect of these changes on student performance. Therefore, we examined the effect of the length, timing, and content of the third year surgery rotation on several clerkship and postclerkship performance measures of 487 students from July 1994 to July 1998. In addition, students’ perceptions regarding their ability to understand surgical disease topics were surveyed. The 8-week clerkship (n 5 232) was associated with higher NMBE surgery test scores (510.5 6 6.3 versus 457.4 6 10.0, P < 0.05) resulting in higher final clerkship grades (5.15 6 0.04 versus 4.87 6 0.03, P < 0.05). Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores, the longer clerkship was associated with higher total (70.6 6 0.37 versus 68.8 6 0.50, P < 0.05) and abdominal pain station (81.87 6 0.71 versus 79.54 6 0.73, P < 0.05) OCSE scores. Students rotating on surgery during the second half of third year (n 5 233) had higher NMBE surgery test scores (513.1 6 8.9 versus 460.5 6 11.2, P < 0.05) and final grades (5.17 6 0.03 versus 4.81 6 0.04, P < 0.05). Although the timing of the surgery clerkship did not significantly affect total OSCE scores, students who rotated on surgery in the second half of third year performed significantly better year on the abdominal pain OSCE station (80.47 6 0.92 versus 76.49 6 1.27, P < 0.05). Students who rotated on general surgery (n 5 298) performed significantly better on the NBME surgery test (525.6 6 6.0 versus 459.6 6 9.1, P < 0.05), although this did not significantly affect the final grade. Although general versus subspecialty surgery 1 To whom correspondence should be addressed at P.O. Box 100286, Department of Surgery, University of Florida, Gainesville, FL 32610. Fax: (352) 338-9809.
0022-4804/99 $30.00 Copyright © 1999 by Academic Press All rights of reproduction in any form reserved.
rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station (81.21 6 0.91 versus 78.17 6 0.32, P < 0.05). The length, timing, and content of the third year surgery rotation had no significant effect on performance on the oral examination. Students who had a 6-week clerkship and students who lacked exposure to general surgery felt their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery. The length, timing, and content of the surgery clerkship affect some clerkship performance measures and student perceptions of their understanding of surgical disease topics. While cognitive differences related to clerkship length are no longer detectable at the end of the third year of medical school, differences related to the acquisition of some clinical skills persist after the surgery clerkship. © 1999 Academic Press Key Words: clerkship; NMBE surgery test; USMLE step 2, OSCE. INTRODUCTION
The emphasis on a generalist professional education has led to shortening the third year surgery clerkship in the curricula of many medical schools (Fig. 1). Studies examining the effect of clerkship characteristics, such as clerkship length, on student performance have produced conflicting results [1– 4]. The differences between these studies may be due to variations in (1) the classification of clerkship variables (i.e., 12 weeks versus 8 weeks), (2) the indicators of educational outcome, and (3) school-specific differences with respect to students, curricula, and educational methods. Changes in the curriculum at the University of Florida prompted an analysis of the effect of the length, timing, and
106
107
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
TABLE 1 The Curricular Changes in the Third Year at the University of Florida Length (weeks)
FIG. 1. The change in surgery and primary care (family practice, internal medicine, pediatrics, and obstetrics/gynecology) clerkship length in U.S. medical schools. Source: The American Association of Medical Colleges (AAMC) Clerkship Directory.
content of the third year surgery rotation on several student performance measures over a 4-year period that spanned the recent curricular changes. METHODS The performance of 487 students from July 1994 to July 1998 was assessed. In 1996 the curriculum at the University of Florida underwent significant changes that resulted in a shortening of the third year surgery clerkship from 8 to 6 weeks (Table 1). Prior to this curtailment, the 8-week surgery rotation consisted of two 4-week rotations on the surgical services listed in Table 2 (1994 –1996). In July of 1996 students began rotating for 6 weeks on the surgical services listed in Table 2 (1996 –1998). Clerkship performance measures examined were the National Board of Medical Examiners (NBME) surgery test (25% of the final grade), an oral examination (25% of the final grade), and the surgery final grade. The ward evaluation of student performance was not part of this analysis but it composes 50% of the final grade. The NBME surgery test is a 2-h 120-item multiple-choice examination administered on the last day of the clerkship. NBME total scores have been shown to have a reliability of 0.80 [5]. The format of the oral examination consists of a faculty member questioning the student regarding the evaluation and management of two topics from a list of common surgical problems (30 min per topic). Ward evaluations are made by faculty utilizing a clinical evaluation form consisting of 10 items considered to be important components of professional competence (i.e., fund of knowledge, problem-solving, clinical judgement, etc.). Postclerkship performance measures included the United States Medical Licensing Examination (USMLE) step 2 and an Objective Structured Clinical Examination (OSCE). The OSCE is completed by students at the end of third year to provide a performance-based assessment of knowledge and skills acquired during the third year clinical rotations. The OSCE consists of 9 to 12 stations (abdominal pain, back pain, and shortness of breath, etc.) involving standardized patients who evaluate student performance using a checklist format. In addition, at the end of the third year, the students are required to take the USMLE step 2. The USMLE step 2 is a 2-day multiplechoice question (MCQ) examination that stresses integration of information across the medical disciplines. USMLE step 2 has been shown to have a reliability of 0.97 [5]. The manner in which USMLE step 2 scores are reported to the institution did not permit an analysis of the effect of the timing and content of the surgery clerkship on this performance measure. Only the first examination scores were
Clerkship
1994–1996
Medicine Anesthesiology Geriatrics Neurology ACLS Primary Care Comm. Medicine Pediatrics Ob-Gyn Psychiatry Surgery
8 1
1996–1998 M A G N A
2
12
6
12
8 8 7 8 44
6 6 6 6 44
Note. MAGNA, medicine, anesthesiology, geriatrics, neurology, and advanced cardiac life support (ACLS).
used for analysis for students who required retesting for any performance parameter. To examine of the effect of clerkship timing on student performance, students were divided into those students who rotated on surgery during the first half of the third year of medical school (July to January) versus those students who rotated on surgery during the last half of the academic year (January to July). Clerkship content was categorized as general surgery if students rotated on general surgery for the entire 6-week rotation or for 4 weeks as part of the 8-week surgery rotation (i.e., 4 weeks subspecialty service and 4 weeks general surgery service). Students who lacked any exposure to a general surgery service were categorized as subspecialty rotation students. Surgery rotations (i.e., subspecialty versus general) were determined by a lottery system. Data was analyzed by the Student t test and a P value of 0.05 was used for significance.
TABLE 2 Structure of the Surgery Clerkship Surgery service
1994–1996 a
1996–1998 b
General surgery 1 General surgery 6 General surgery 8 VA general surgery General surgery (JAX) Vascular VA vascular Vascular (JAX) Plastics Plastics (JAX) Transplant Urology Urology (JAX) Pediatric surgery
X X
X X X X X X X
a b
Two 4-week rotations. One 6-week rotation.
X X X X X X X X X X X
X X X X
108
JOURNAL OF SURGICAL RESEARCH: VOL. 84, NO. 1, JUNE 1, 1999
FIG. 2. (A) The effect of clerkship length on clerkship performance measures. Data represent mean 6 SEM. *P , 0.05. (B) The effect of clerkship length on USMLE step 2 total score and surgery subsection score. (C) The effects of clerkship length on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
RESULTS
Clerkship Length and Student Performance Figure 2A demonstrates that the 8-weeks clerkship (n 5 232) produced higher NMBE surgery test scores (510.5 6 6.3 versus 457.4 6 10.0, P , 0.05) resulting in higher final clerkship grades (5.15 6 0.04 versus 4.87 6 0.03, P , 0.05), but it had no significant effect on oral examination scores. Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores (Fig. 2B), the longer clerkship was associated with higher total (70.6 6 0.37 versus 68.8 6 0.50, P , 0.05) and abdominal pain station (81.87 6 0.71 versus 79.54 6 0.73, P , 0.05) OCSE scores assessed at the end of the third year of medical school (Fig. 2C).
Clerkship Timing and Student Performance Figure 3A shows that students rotating on surgery during the second half of third year (n 5 233) had higher NMBE surgery test scores (513.1 6 8.9 versus 460.5 6 11.2, P , 0.05) and final grades (5.17 6 0.03 versus 4.81 6 .04, P , 0.05), but the timing of the surgery clerkship had no significant effect on oral examination scores. Although the timing of the surgery clerkship did not significantly affect total OSCE scores, students who rotated on surgery in the second half of third year performed significantly better on the abdominal pain OSCE station (80.47 6 0.92 versus 76.49 6 1.27, P , 0.05) than students who rotated on surgery in the first half of the third year (Fig. 3B).
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
109
FIG. 3. (A) The effect of clerkship timing on clerkship performance measures. *P , 0.05. (B) The effects of clerkship timing on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
Clerkship Content and Student Performance Students who rotated on general surgery (n 5 298) performed significantly better on the NBME surgery test (525.6 6 6.0 versus 459.6 6 9.1, P , 0.05), although general versus subspecialty rotation did not significantly affect performance on the oral surgery examination or the final grade (Fig. 4A). Although general versus subspecialty surgery rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station (81.21 6 0.91 versus 78.17 6 0.32, P , 0.05) (Fig. 4B). Survey Analysis Response to the student survey was 68% and there was no difference in the response rate with respect to clerkship length or content (general versus subspecialty). Students who had a 6-week clerkship or students who lacked exposure to general surgery felt that their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery (Table 3). DISCUSSION
The national agenda to increase the number of generalist physicians has led to shortening the surgery clerkship in the curricula of many medical schools (Fig. 1). Studies examining the effect of clerkship curtailment on student performance have produced conflicting results. Smith et al. found that shortening the obstetrics and gynecology clerkship from 8 weeks to 6 weeks had no effect on the NMBE obstetrics and gynecology subject test performance [1]. On the other hand,
investigators from the National Board of Medical Examiners found that students from 32 schools with a 12-week surgery clerkship performed significantly better on the NBME surgery test and USMLE step 2 than students from 25 schools with an 8-week surgery clerkship [5]. This study failed, however, to address whether clerkship curtailment from 8 weeks to 6 weeks produced similar differences in student performance and this study only assessed cognitive measures of clinical competency. In addition, data from a national study may have limited applicability to individual medical schools because of school-specific curricular factors. Furthermore, little data exist regarding the relationship between clerkship variables and postclerkship measures of student performance. In our study, the shortening of the surgery clerkship from 8 weeks to 6 weeks significantly affected student performance on the NBME surgery test. Since clerkship curtailment had no effect on performance on the oral examination or ward (faculty) evaluations (data not shown), the effect of clerkship duration on final grade was solely the result of the lower NBME surgery test scores. Although clerkship length had no significant effect on USMLE step 2 total or surgery subsection scores, the longer clerkship was associated with higher total and abdominal pain station OSCE scores assessed at the end of the third year of medical school. Thus, it appears that cognitive differences associated with the change in clerkship length are no longer detectable at the end of the third year. Several possible explanations may account for this finding. It is conceivable that differences in NBME surgery test scores associated with shortening the surgery clerkship by 2 weeks were simply attributable to the 8-week students benefiting from the extra 2 weeks of preparatory study time. Alternatively, the differ-
110
JOURNAL OF SURGICAL RESEARCH: VOL. 84, NO. 1, JUNE 1, 1999
FIG. 4. (A) The effect of clerkship content (general surgery versus subspecialty rotation) clerkship performance measures. *P , 0.05. (B) The effects of clerkship content (general surgery versus subspecialty rotation) on postclerkship total and abdominal pain station OSCE scores. *P , 0.05.
ences in NBME surgery test scores maybe related to actual learning deficiencies associated with a shortened clerkship. The inability to detect an effect of clerkship length on postclerkship USMLE step 2 total or surgery subsection scores suggests that with time students overcome the detrimental effects of the shortened clerkship on cognitive learning. Poor student performance on the NBME surgery test may have influenced students to prepare more for the USMLE step 2 examination. The longer clerkship was, however, associated with higher total and abdominal pain station OSCE scores asTABLE 3 Response to Student Survey Mean student response a Surgical topics
8 weeks
Fluid and electrolytes Wound healing Trauma Breast Acute abdomen Gastrointestinal hemorrhage Common vascular problems Perioperative care
2.8 6 1.0
a
6 weeks
General
Specialty
b
2.8 6 1.1
2.6 6 1.1*
3.0 6 1.0
1.5 6 0.9* 1.7 6 0.9* 2.0 6 1.3* 2.2 6 1.0 2.0 6 0.9*
2.2 6 1.0 2.6 6 1.0 2.7 6 1.2 2.5 6 1.2 2.6 6 1.2
1.7 6 1.0* 1.8 6 0.9* 2.5 6 1.2* 2.1 6 1.0* 2.2 6 1.0*
2.3 6 1.2 2.5 6 1.0 2.9 6 1.1 2.9 6 1.2 2.6 6 1.3
2.6 6 1.3
2.6 6 1.1
2.8 6 1.5
2.4 6 1.0
2.1 6 0.9*
2.5 6 1.1
2.1 6 0.9*
2.5 6 1.1
Using the Likert-type scale below, please respond to the statement: “I feel my surgery rotation adequately prepared me to understand the following surgical topics. . .” Strongly Agree 1 2 3 4 5 Strongly Disagree. b Data 5 mean 6 SEM. * P , 0.05 by t test.
sessed at the end of the third year of medical school, suggesting that clinical skills deficits exist postclerkship. The lack of an OSCE immediately following the surgery clerkship makes it impossible to determine precisely whether these clinical skill deficits are persistent or develop de novo postclerkship. Our study demonstrates that students rotating on surgery during the second half of third year had higher NMBE surgery test scores resulting in higher final grades. Other studies have also demonstrated that clerkship timing affects educational outcome [2, 3]. Our findings probably reflect knowledge gained by the student on previous clerkships, particularly the internal medicine clerkship, which can be applied to the NBME surgery test. This finding may also provide justification for “handicapping” students depending upon when they rotate on surgery during the third year of medical school. This handicapping may have particular importance for those students competing for a surgical residency. It is important to note, however, that the order of clerkship rotation was determined by lottery, minimizing the bias of student selection to this difference (i.e., students with surgical career interests selecting surgery at a particular time to maximize their academic performance). In this study, students who rotated on general surgery performed significantly better on the NBME surgery test but the type of rotation did not significantly affect the final grade. Although general versus subspecialty surgery rotation did not significantly affect total OSCE scores, students rotating on general surgery performed significantly better on the abdominal pain OSCE station. The effect of the clinical setting of the surgery rotation on student performance has also been previously studied [6]. Poenaru et al. found no differ-
LIND ET AL.: SURGERY CLERKSHIP AND PERFORMANCE
ence between students rotating on a general surgical and subspecialty service in several educational parameters. They concluded that effective undergraduate education could occur in a subspecialty setting as long as the educational process is deliberate, structured, and objective-based. The effect of a lack of a general surgery rotation on performance on the abdominal pain OSCE maybe due to a lack of exposure to the common general surgical problem in the subspecialty setting. The fact that the length, timing, and content of the surgery clerkship did not influence performance on the surgery oral examination is not surprising. Other studies have found little correlation between oral examinations and other performance measures [7]. In general, oral examinations may be unreliable instruments to discriminate between students [8]. Interestingly, our survey data complement our educational performance data. Students who had a 6-week clerkship or students who lacked exposure to general surgery felt their surgery rotation failed to prepare them to understand a number of surgical topics as well as students who had an 8-week clerkship or students who rotated on general surgery (Table 3). There are some limitations to this study. Although there were no obvious differences in instructional methods between the study groups, other factors such as differences in the use of ambulatory setting for teaching or differences in student-faculty contact between the groups may account for some of the differences detected in student performance. Alternatively, the differences in student performance may be due to educational differences that existed prior to the surgery clerkship. Analysis of student performance prior to the surgery clerkship (i.e., Medical Colleges Aptitude Test (MCAT) scores, Grade Point Averages, and USMLE step 1 scores) shows no difference between the study groups. In fact, USMLE step 1 scores at the University of Florida have steadily increased in the past 5 years [9]. In addition, our study ignores nonsurgical curricular changes that have occurred at the University of Florida (Table 1). The shortening of the pediatrics and obstetrics and gynecology clerkships may impact student performance measures at the end of the third year that assess clinical competence across many
111
medical disciplines (i.e., total OSCE score and total USMLE step 2 scores). Finally, caution should also be used before extrapolating our data to other institutions. In summary, the length, timing, and content of the surgery clerkship affect some clerkship performance measures and student perceptions of their understanding of surgical disease topics. While cognitive differences related to clerkship length are no longer detectable at the end of the third year of medical school, differences related to the acquisition of some clinical skills persist after the surgery clerkship. Data on the postgraduate educational consequences of shortening the surgery clerkship are sorely required. REFERENCES 1.
2.
3.
4.
5.
6.
7.
8.
9.
Smith, E. R., Dinh, T. V., and Anderson, G. A decrease from 8 to 6 weeks in obstetrics and gynecology clerkship: Timing on medical student’s cognitive knowledge. Obstet. Gynecol. 86: 458, 1995. Whalen, J. P., and Moses, V. K. The effect on grades of the timing and site of third year internal medicine clerkships. Acad. Med. 65: 708, 1990. Baciewicz, F. A., Arent, L., Weaver, M., Yeasting, R., and Thomford, N. R. Influence of clerkship structure and timing on individual student performance. Am. J. Surg. 159: 265. Gary, N. E., and Rosevar, G. C. Effect of reduction in length of the third clerkships on students’ academic performance. J. Med. Educ. 63: 406, 1988. Ripkey, D. R., Case, S., and Swanson, D. B. Predicting performances on the NBME surgery subject test and USMLE step 2: The effects of surgery clerkship timing length. Acad. Med. 72(10): S31, 1997. Poenaru, D., Davidson, L., Donnely, M., and Tekain, A. Is a mandatory general surgery rotation necessary in the surgical clerkship? Am. J. Surg. 175: 515, 1998. Amiel, G. E., Tann, M., Krausz, M. M., Bitterman, A., and Cohen, R. Increasing examiner involvement in an objective structured clinical examination by integrating a structured oral exam. Am. J. Surg. 173: 546, 1997. Stillman, R. M., Lane, K. M., Beeth, S., and Jaffee, B. M. Evaluation of the student. Improving validity of the oral examination. Surgery 93: 439, 1983. Watson, R. T., Suter, E., Romrell, L. J., Harmon, E. M., Rooks, L., and Neims, A. H. Moving a graveyard: How one school prepared the way for continuous curriculum renewal. Acad. Med. 73: 948, 1998.