- Email: [email protected]
The surgical clerkship experience: Self, departmental, and institutional assessment
ASSOCIATION FOR SURGICAL EDUCATION
The Surgical Clerkship Experience: Self, Departmental, and Institutional Assessment Jocelyn D. Ten Haken, PhD, Judith G. Calhoun, PhD, Kenneth A. Frank, BA, Linda C. Youmans; BA, Gerald B. Zelenock, MD, Ann Arbor,Michigan
The purpose of this study was to investigate to what extent students' assessments of their clerkship experiences varied in relation to perceptions of self-improvement, the time of year in which the clerkship experience took place, and the origin of the assessment device (departmental versus institutional). All students enrolled in each of six surgery clerkship rotations over the course of 1 academic year ( 2 2 4 students) were administered two questionnaires on completion of the clerkship. One questionnaire was institutionally developed and supported, whereas the other was departmentally sponsored. Results of multiple regression analyses indicated that ( 1 ) when students felt their medical skills had improved as a result of clerkship participation, the clerkship was considered of high quality; ( 2 ) students' perceived improvement in communication skills decreased as the academic year progressed; and ( 3 ) students did not alter their responses to the questionnaires based on whether the instrument was institutionally or departmentally sponsored.
valuation of the student experience during required E clinical clerkships is an essential responsibility of individual teachers as well as departmental and medical school educational administrators. This responsibility includes conducting a comprehensive evaluation in which the viewpoints of students, clinical faculty, and medical school administration are considered [1-3]. However, very few studies have included student assessment as an actual evaluation component of student performance [4]. Concerns that such assessments may be influenced by who was requesting the information as well as students' previous academic experiences prompted this investigation. Previous research on student self- and peer assessment From the Departments of Surgery and Postgraduate Medicine and Health ProfessionsEducation, The Universityof Michigan Medical School,Ann Arbor, Michigan. Requestsfor reprintsshouldbe addressedto JocelynD. Ten Haken, PhD, LearningResourceCenter,The Universityof Michigan Medical School,1135East CatherineStreet,Ann Arbor,Michigan48109-0726. Presented at the SeventhAnnual Meeting of the Associationfor Surgical Education,Williamsburg,Virginia,April 8-1l, 1987.
has investigated the relationship between such assessments and cognitive measures, and expert assessment of student performance [5-8]. These studies found that students' perceptions of their own performances increased over time and that students' perceptions were generally correlated with experts' evaluations. In addition, students' self-evaluations did not correlate with objective measures of cognitive performance. Previous research, however, has not addressed the relationship between students' perceptions of self-improvement and their perceptions of the quality of the clerkship, The goal of this study was to investigate to what extent students' assessments of their clerkship experiences vary in relation to their perceptions of self-improvement and the time of year in which the clerkship experience takes place. The quality of the clerkship from an institutional perspective and departmental viewpoint was likewise assessed, Specific research questions included the following: (1) What is the relationship between students' perceptions of improvement of their clinical skills and ratings of clerkship quality and value? (2) To what extent do students' evaluations of clerkship experiences vary as the academic year progresses? (3) Is there a difference in students' ratings of the quality or value of the clerkship depending on whether the students complete an institutional or departmental evaluation questionnaire? METHODS Medical students at The University of Michigan are required to participate in an 8 week general surgery clerkship during their junior year. In addition, they must elect two 4-week experiences in surgical subspecialties; the latter also includes offerings from the departments of anesthesiology, otolaryngology, and ophthalmology. All 224 students enrolled in each of the 6 required general surgery clerkship rotations over the course of 1 academic year were administered two questionnaires on completion of the clerkship. The first questionnaire was developed in 1984 by an interdepartmental committee of clerkship coordinators and was institutionally supported by the medical school administration for use in all required clerkships, including surgery, internal medicine, psychiatry, obstetrics and gynecology, and pediatrics. The second questionnaire was developed and supported by the surgery department and used since 1976 for the required surgery clerkship. The medical school questionnaire was based in part on the knowledge and skills on which faculty and residents from all five departments evaluated students. The questionnaire consisted of 22 Likert-type items. Twelve of the items related to the clinical skills and knowledge that
THE AMERICAN JOURNAL OF SURGERY VOLUME156 SEPTEMBER 1988 155
TEN HAKEN ET AL
TABLE I Content of Questionnaire Items Categorized by Factor Analysis
y = .160374(MS)* - .08221(CS) + .10972 (T) + 1.2342
Factor
Skills and Clerkship Quality
I
Communication skills Patient interviewing Communication skills Written skills Professionalism Interpersonal skills with patients Interpersonal skills with physicians Quality of clerkship Goals clearly explained Faculty as teachers Residents as teachers Achievement of goals Instructors provided motivation Guidance for literature study Increased clinical knowledge Time for independent study Performed up to potential Excellent clerkship
Medical skills Fund of knowledge History taking Physical examination Differential diagnosis Diagnostic/therapeutic planning Oral presentations
students might have obtained while participating in the clerkship. These items have been used since 1977 to evaluate student performance in all clerkships. Students indicated their perception of self-improvement on a scale from 1 to 5 (no improvement to major improvement) on each of these items. In addition, 10 items pertained to various aspects of the clerkship itself, surveying student assessment of the teaching skills of residents and faculty, clarity of clerkship goals and objectives, and the facilitation of both independent and general student learning. Students indicated on a scale of 1 to 5 their level of agreement with each of the statements, with 1 representing strong disagreement and 5 representing strong agreement. A reliability measure based on paired items within this group was determined to be 0.60 to 0.70 across the five clerkships (internal medicine, surgery, pediatrics, psychiatry, and obstetrics and gynecology) using the instrument. The reliability coefficient for surgery alone was 0.61. Finally, students were asked to outline the types of academically unproductive activities in which they participated as well as the three best and three worst aspects of the clerkship experience. For purposes of data analysis, an oblique rotation factor analysis yielded three distinct factors from the institutional questionnaire. These factors were based on loadings greater than 0.4. Factor 1, which was composed of six items, represented students' perceptions of their improvement in communication skills. Factor 2, consisting of 10 items, formed a factor representing perception of the overall quality of the clerkship. The third factor, comprising six items, represented students' perceptions of 156
THE A M E R I C A N J O U R N A L OF SURGERY
Figure 1. Regression equation denoUng variables affecting students' perceptions of the overall quality of the clerkship. CS = perception of improvement of communicaUon skills (factor 1); MS = p e r c e p t i o n of i m p r o v e m e n t of medical skills (factor 3); T = t i m e period of clerkship; and y = raUngs of overall quality of clerkship (factor 2). n = 133; R 2 = 0.43077. Asterisk indicates p < 0 . 0 0 1 .
improvement in their medical skills. Each item loaded on one and only one factor. Table I delineates the content of the items which constituted each of the three factors. The second questionnaire, developed and supported by the surgery department, asked students to rate 11 aspects of the clerkship on a scale from 1 (poor) to 5 (outstanding). The items on this questionnaire addressed clerkship orientation, rounds, conferences, patient population and exposure, recommended texts, and the operating room experience. The questionnaire also provided students with the opportunity to delineate the best and worst experiences of the surgery clerkship. A content analysis of the open-ended items, which asked students to list the positive and negative aspects of the clerkship on both the institutional and departmental questionnaires, was undertaken to identify possible differences and similarities using key word categorization. The use of specific names, recommendations, and complaints about policy were summarized. RESULTS Results were assessed with regard to the specific research questions. Question 1: What is the relationship between student perceptions of their improvement in clinical skills and ratings of clerkship quality and value? Multiple regression analysis was used to determine the relationship among students' overall ratings of the clerkship (factor 2), perceptions of improvement of communication skills (factor 1), and perceptions of improvement of medical skills (factor 3). The time period in which the clerkship took place was included as a covariate in the regression equation. The regression indicated that the coefficient associated with students' perceptions of improvement in medical skills was significant at p <0.001 (Figure 1). The coefficient associated with students' perceptions of their improvement in communication skills was not significant. A separate regression was run for each of the six time periods in which a clerkship took place. Communication skills (factor 1) and medical skills (factor 3) were used as independent variables, with students' evaluations of the overall quality of the clerkship (factor 2) as the dependent variable. For each time period, the coefficient representing perceptions of improvement in medical skills was significant at p <0.05. Conversely, the coefficient representing perceptions of improvement in communication skills was not significant for any of the six time periods. Question 2: To what extent do student evaluations of their clerkship experiences vary as the a c a d e m i c year progresses? Regression analyses were performed using the time period in which the clerkship took place as
VOLUME 156
SEPTEMBER 1988
SURGICAL CLERKSHIP EXPERIENCE
TABLE III
TABLE II The Effect of Time Period of Clerkship Rotation on Factors 1, 2, and 3 Factor Communication skills (factor 1) Medical skills (factor 3) Overall clerkship evaluation (factor 2)
n
r
F
139
--0.30409
13.95"
144
0.03629
0.18
135
0.16217
3.95
* p < 0.001. F = test of significance.
an independent variable in each of three equations with the three factors as dependent variables (Table II). The time period that the clerkship was elected proved to be a significant predictor of students' perceptions of improvement in communications skills. The negative correlation between these two variables indicated that with each successive time period, students' communication skills improved less. Time period was not a significant predictor of medical skills improvement or of overall ratings of the quality of the clerkship. Question 3: Is there a difference in student ratings of the quality or value of the clerkship depending upon whether the students complete an institutional or departmental evaluation questionnaire? Items sim-
ilar in content were identified from each questionnaire and matched to provide a basis of comparison between the institutional and departmental questionnaires. Five Likert-type items and two open-ended questions were matched for this comparison (Table III). A Student's t statistic was used to compare the means of the matched items. Paired comparisons could not be made because individual student identifications were not generally established on each questionnaire. Comparisons were made for all periods combined and for each of the six separate periods. "Clarity of clerkship goals" (comparison 3) was consistently rated higher on the institutional questionnaire than on the departmental questionnaire. This relationship held true for each time period and was significant for the mean reflecting all time periods combined (Table III). The means reflecting students' perceptions of the "facult y a s teachers" on the institutional questionnaire were higher, although not significantly, than those on the departmental questionnaire. This relationship was consistent for each period as well as for all periods combined. The only other pair of matched items that showed a significant difference was comparison 5, "guidance for study of literature" versus "recommended clerkship readings." Mean values on the institutional questionnaire were lower than the departmental questionnaire for each period and for all periods combined. The relationship was significantly different for the means reflecting periods 2 and 4, and for the mean reflecting all time periods combined. A content analysis of students' comments on the openended questionnaire items indicated that students tended to use specific names in discussing faculty or residents
Comparison of Corresponding Institutional (I) Versus Departmental (D) Questionnaire items for All Six Clerkship Rotations Combined Comparison
n
Mean
t
Residents (I) Residents (D)
142 117
3.47 3.44
0.21630
Independent study time {I) Outside reading time (D)
144 118
2.43 2.61
1.0587
Goals clearly explained (I) Clerkship orientation (D)
142 120
3.52 3.23
2.5411 *
Faculty as teachers (I) Faculty teaching rounds (D)
142 114
3.64 3.31
0.1587
Guidance for study of literature (I) Recommended clerkship readings (D)
144 109
2.95 3.36
3.3903*
* p < 0.01. t = test of significance.
with similar frequency on both questionnaires. The most frequently cited positive aspects of the clerkship for both questionnaires included the patient interaction, the opportunity to learn surgical skills, and the quality of some of the departmental staff. The most frequently cited negative aspects included the lack of time for independent study and the condescending attitude toward students by some of the hospital ward staff. Students did not issue any more calls for policy change on the departmental questionnaires than on the institutional questionnaires. There did seem to be a marked difference between the students' attitudes towards the question on the institutional questionnaire: "If you checked 'Scut Work' please describe what you mean by the term." Many described the nature of scut work but then added that they were not complaining, since this type of work had to be done, recognizing the perspective of the person who may read the questionnaire. COMMENTS A comprehensive assessment of clinical clerkships is of concern to individual faculty members, clinical departments, and the medical school administration. However, a concern that students' perceptions of the clerkship might vary depending on who mandated the evaluation, the time period during which the clerkship was elected, and types of activities required during the clerkship prompted this investigation. The results of this study provide valuable information regarding students' perceptions of the quality of a clerkship relative to their perceptions of the skills and knowledge obtained from clerkship participation. Students appear to quickly reach a plateau in what they perceive to be improvement in their communication skills over the course of an academic year. Since students participate in a variety of clerkships during their third year, they have ample opportunity, regardless of specialty, to hone their communication skills. Therefore, students who participated in the surgery clerkship later in the year did not see as much improvement in communication skills as did students who participated early in the academic year.
THE AMERICAN JOURNAL OF SURGERY
VOLUME 156
SEPTEMBER 1988
157
TEN HAKEN ET AL
Despite the plateau effect in communication skills, students believed their medical skills continued to increase throughout the year. When students thought their medical skills had improved as a result of the clerkship, the clerkship was considered of high quality. Restated, when students were allowed the opportunity to act as a physician, that is, obtain patients' histories and perform physical examinations, construct a differential diagnosis, and outline and participate in a therapeutic plan, they assigned more value to the clerkship experience and hence, rated the clerkship higher than students who experienced less actual patient contact. Students did not alter their responses to clerkship evaluation questionnaires based on whether the instrument was institutionally or departmentally sponsored. Our original bias that students might respond more favorably to a departmentally administered questionnaire proved unfounded. In fact, the medical school assessment consistently rated "faculty as teachers" and "clarity of clerkship goals" higher than did the departmental evaluation. The two comparisons which showed significant differences, "guidance for study of the literature" with "recommended clerkship readings" and "goals clearly explained" with "clerkship orientation," may have resuited from the lack of equivalence between the two pairs. The departmental questionnaire asked very specifically about the readings used to supplement the clerkship, whereas the institutional questionnaire asked about behavior on the part of the instructors to guide students in their literature study. Likewise, the institutional questionnaire addressed whether the goals of the clerkship had been explained to students, whereas the departmental questionnaire asked a global question about the orientation, which could encompass logistics, timing, and so on.
The comments that students included as responses to the open-ended items on both questionnaires addressed similar issues and listed the same complaints regarding the clerkship. Students were equally candid on both questionnaires, indicating lack Of concern for how the information would eventually be used either departmentally or institutionally. There was a significant relationship between perceived improvement of skills required for direct patient contact and students' ratings of the quality of the clerkship. Moreover, previous research has documented a strong correlation between surgery students' perceptions of their own improvement in medical skills and experts' opinions of improvement in these same skills [6]. Thus, this perception of improvement in medical skills on the part of the students may indeed reflect an actual improve-
158
THE AMERICAN JOURNAL O F S U R G E R Y
ment in medical skills. Students clearly defined their personal objectives for clerkship participation as including patient care activities. This study showed that if those objectives are met, the students believe their time has been well spent. The converse is equally true. Students do not think their time is well spent if their clerkship experiences do not include meaningful patient care activities. Clerkship coordinators must be prepared to delineate for students the total value of all clerkship experiences, as well as specifically outline the full range of clerkship objectives. Students who are unaware of these objectives or who are unwilling to accept them as legitimate will continue to underrate the value of the clerkship as a whole. Surgical educators and medical school administrators alike should be heartened by the findings of this study. That assessing clerkships and value assigned to a given experience can be appropriately assigned by either departmental or institutional groups is encouraging but contrary to our preconceived biases. Further, that student perception of the value of a given experience depends in large part on the further development of clinical skills and is not correlated with further development of social and communication skills is important. Finally, that the time of year that the clerkship is elected affects neither student performance nor perception of the clerkship's inherent value is of importance to course directors and suggests that major changes in clerkship design and educational philosophy need not be dictated by the academic calendar, REFERENCES 1. Klintbert IG, Givner N. The comparability of clerkship experiences in a variety of locations. J Med Educ 1977; 52: 518-20. 2. Miller JM, Smith IK, Sosnowski JR, Hester LL. Evaluation of student performance in an obstetrics and gynecology clerkship. J Reprod Med 1982; 27: 443-6. 3, LaPalio LR, Filling CM, Engel JD, Ways PO. Multiple strategies for studying medical clerkship experiences: a case study. J Med Educ 1983; 58: 328-34. 4. Arnold L, Willoughby TL, Calkins EV, Gammon L, Eberhart G. Use of peer evaluation in the assessment of medical students. J Med Educ 1981; 56: 35-42. 5. Morton JB, McBeth WAAG. Correlations between staff, peer and self-assessments of fourth-year students in surgery. J Med Educ 1977; 11: 167-70. 6. Arnold L, Willoughby TL, Calkins EV. Self-evaluation in undergraduate medical education: a longitudinal perspective. J Med Educ 1985; 60: 21-8. 7. Henbest RS, Fersen GS. Preliminary study at the Medical University of Southern Africa on student self-assessment as a means of evaluation. J Med Educ 1985; 60: 66-7. 8. Kegel-Flom P. Predicting supervisor, peer and self ratings of intern performance. J Med Educ 1975; 50: 812-5.
VOLUME 156 SEPTEMBER 1988
The Surgical Clerkship Experience: Self, Departmental, and Institutional Assessment Jocelyn D. Ten Haken, PhD, Judith G. Calhoun, PhD, Kenneth A. Frank, BA, Linda C. Youmans; BA, Gerald B. Zelenock, MD, Ann Arbor,Michigan
The purpose of this study was to investigate to what extent students' assessments of their clerkship experiences varied in relation to perceptions of self-improvement, the time of year in which the clerkship experience took place, and the origin of the assessment device (departmental versus institutional). All students enrolled in each of six surgery clerkship rotations over the course of 1 academic year ( 2 2 4 students) were administered two questionnaires on completion of the clerkship. One questionnaire was institutionally developed and supported, whereas the other was departmentally sponsored. Results of multiple regression analyses indicated that ( 1 ) when students felt their medical skills had improved as a result of clerkship participation, the clerkship was considered of high quality; ( 2 ) students' perceived improvement in communication skills decreased as the academic year progressed; and ( 3 ) students did not alter their responses to the questionnaires based on whether the instrument was institutionally or departmentally sponsored.
valuation of the student experience during required E clinical clerkships is an essential responsibility of individual teachers as well as departmental and medical school educational administrators. This responsibility includes conducting a comprehensive evaluation in which the viewpoints of students, clinical faculty, and medical school administration are considered [1-3]. However, very few studies have included student assessment as an actual evaluation component of student performance [4]. Concerns that such assessments may be influenced by who was requesting the information as well as students' previous academic experiences prompted this investigation. Previous research on student self- and peer assessment From the Departments of Surgery and Postgraduate Medicine and Health ProfessionsEducation, The Universityof Michigan Medical School,Ann Arbor, Michigan. Requestsfor reprintsshouldbe addressedto JocelynD. Ten Haken, PhD, LearningResourceCenter,The Universityof Michigan Medical School,1135East CatherineStreet,Ann Arbor,Michigan48109-0726. Presented at the SeventhAnnual Meeting of the Associationfor Surgical Education,Williamsburg,Virginia,April 8-1l, 1987.
has investigated the relationship between such assessments and cognitive measures, and expert assessment of student performance [5-8]. These studies found that students' perceptions of their own performances increased over time and that students' perceptions were generally correlated with experts' evaluations. In addition, students' self-evaluations did not correlate with objective measures of cognitive performance. Previous research, however, has not addressed the relationship between students' perceptions of self-improvement and their perceptions of the quality of the clerkship, The goal of this study was to investigate to what extent students' assessments of their clerkship experiences vary in relation to their perceptions of self-improvement and the time of year in which the clerkship experience takes place. The quality of the clerkship from an institutional perspective and departmental viewpoint was likewise assessed, Specific research questions included the following: (1) What is the relationship between students' perceptions of improvement of their clinical skills and ratings of clerkship quality and value? (2) To what extent do students' evaluations of clerkship experiences vary as the academic year progresses? (3) Is there a difference in students' ratings of the quality or value of the clerkship depending on whether the students complete an institutional or departmental evaluation questionnaire? METHODS Medical students at The University of Michigan are required to participate in an 8 week general surgery clerkship during their junior year. In addition, they must elect two 4-week experiences in surgical subspecialties; the latter also includes offerings from the departments of anesthesiology, otolaryngology, and ophthalmology. All 224 students enrolled in each of the 6 required general surgery clerkship rotations over the course of 1 academic year were administered two questionnaires on completion of the clerkship. The first questionnaire was developed in 1984 by an interdepartmental committee of clerkship coordinators and was institutionally supported by the medical school administration for use in all required clerkships, including surgery, internal medicine, psychiatry, obstetrics and gynecology, and pediatrics. The second questionnaire was developed and supported by the surgery department and used since 1976 for the required surgery clerkship. The medical school questionnaire was based in part on the knowledge and skills on which faculty and residents from all five departments evaluated students. The questionnaire consisted of 22 Likert-type items. Twelve of the items related to the clinical skills and knowledge that
THE AMERICAN JOURNAL OF SURGERY VOLUME156 SEPTEMBER 1988 155
TEN HAKEN ET AL
TABLE I Content of Questionnaire Items Categorized by Factor Analysis
y = .160374(MS)* - .08221(CS) + .10972 (T) + 1.2342
Factor
Skills and Clerkship Quality
I
Communication skills Patient interviewing Communication skills Written skills Professionalism Interpersonal skills with patients Interpersonal skills with physicians Quality of clerkship Goals clearly explained Faculty as teachers Residents as teachers Achievement of goals Instructors provided motivation Guidance for literature study Increased clinical knowledge Time for independent study Performed up to potential Excellent clerkship
Medical skills Fund of knowledge History taking Physical examination Differential diagnosis Diagnostic/therapeutic planning Oral presentations
students might have obtained while participating in the clerkship. These items have been used since 1977 to evaluate student performance in all clerkships. Students indicated their perception of self-improvement on a scale from 1 to 5 (no improvement to major improvement) on each of these items. In addition, 10 items pertained to various aspects of the clerkship itself, surveying student assessment of the teaching skills of residents and faculty, clarity of clerkship goals and objectives, and the facilitation of both independent and general student learning. Students indicated on a scale of 1 to 5 their level of agreement with each of the statements, with 1 representing strong disagreement and 5 representing strong agreement. A reliability measure based on paired items within this group was determined to be 0.60 to 0.70 across the five clerkships (internal medicine, surgery, pediatrics, psychiatry, and obstetrics and gynecology) using the instrument. The reliability coefficient for surgery alone was 0.61. Finally, students were asked to outline the types of academically unproductive activities in which they participated as well as the three best and three worst aspects of the clerkship experience. For purposes of data analysis, an oblique rotation factor analysis yielded three distinct factors from the institutional questionnaire. These factors were based on loadings greater than 0.4. Factor 1, which was composed of six items, represented students' perceptions of their improvement in communication skills. Factor 2, consisting of 10 items, formed a factor representing perception of the overall quality of the clerkship. The third factor, comprising six items, represented students' perceptions of 156
THE A M E R I C A N J O U R N A L OF SURGERY
Figure 1. Regression equation denoUng variables affecting students' perceptions of the overall quality of the clerkship. CS = perception of improvement of communicaUon skills (factor 1); MS = p e r c e p t i o n of i m p r o v e m e n t of medical skills (factor 3); T = t i m e period of clerkship; and y = raUngs of overall quality of clerkship (factor 2). n = 133; R 2 = 0.43077. Asterisk indicates p < 0 . 0 0 1 .
improvement in their medical skills. Each item loaded on one and only one factor. Table I delineates the content of the items which constituted each of the three factors. The second questionnaire, developed and supported by the surgery department, asked students to rate 11 aspects of the clerkship on a scale from 1 (poor) to 5 (outstanding). The items on this questionnaire addressed clerkship orientation, rounds, conferences, patient population and exposure, recommended texts, and the operating room experience. The questionnaire also provided students with the opportunity to delineate the best and worst experiences of the surgery clerkship. A content analysis of the open-ended items, which asked students to list the positive and negative aspects of the clerkship on both the institutional and departmental questionnaires, was undertaken to identify possible differences and similarities using key word categorization. The use of specific names, recommendations, and complaints about policy were summarized. RESULTS Results were assessed with regard to the specific research questions. Question 1: What is the relationship between student perceptions of their improvement in clinical skills and ratings of clerkship quality and value? Multiple regression analysis was used to determine the relationship among students' overall ratings of the clerkship (factor 2), perceptions of improvement of communication skills (factor 1), and perceptions of improvement of medical skills (factor 3). The time period in which the clerkship took place was included as a covariate in the regression equation. The regression indicated that the coefficient associated with students' perceptions of improvement in medical skills was significant at p <0.001 (Figure 1). The coefficient associated with students' perceptions of their improvement in communication skills was not significant. A separate regression was run for each of the six time periods in which a clerkship took place. Communication skills (factor 1) and medical skills (factor 3) were used as independent variables, with students' evaluations of the overall quality of the clerkship (factor 2) as the dependent variable. For each time period, the coefficient representing perceptions of improvement in medical skills was significant at p <0.05. Conversely, the coefficient representing perceptions of improvement in communication skills was not significant for any of the six time periods. Question 2: To what extent do student evaluations of their clerkship experiences vary as the a c a d e m i c year progresses? Regression analyses were performed using the time period in which the clerkship took place as
VOLUME 156
SEPTEMBER 1988
SURGICAL CLERKSHIP EXPERIENCE
TABLE III
TABLE II The Effect of Time Period of Clerkship Rotation on Factors 1, 2, and 3 Factor Communication skills (factor 1) Medical skills (factor 3) Overall clerkship evaluation (factor 2)
n
r
F
139
--0.30409
13.95"
144
0.03629
0.18
135
0.16217
3.95
* p < 0.001. F = test of significance.
an independent variable in each of three equations with the three factors as dependent variables (Table II). The time period that the clerkship was elected proved to be a significant predictor of students' perceptions of improvement in communications skills. The negative correlation between these two variables indicated that with each successive time period, students' communication skills improved less. Time period was not a significant predictor of medical skills improvement or of overall ratings of the quality of the clerkship. Question 3: Is there a difference in student ratings of the quality or value of the clerkship depending upon whether the students complete an institutional or departmental evaluation questionnaire? Items sim-
ilar in content were identified from each questionnaire and matched to provide a basis of comparison between the institutional and departmental questionnaires. Five Likert-type items and two open-ended questions were matched for this comparison (Table III). A Student's t statistic was used to compare the means of the matched items. Paired comparisons could not be made because individual student identifications were not generally established on each questionnaire. Comparisons were made for all periods combined and for each of the six separate periods. "Clarity of clerkship goals" (comparison 3) was consistently rated higher on the institutional questionnaire than on the departmental questionnaire. This relationship held true for each time period and was significant for the mean reflecting all time periods combined (Table III). The means reflecting students' perceptions of the "facult y a s teachers" on the institutional questionnaire were higher, although not significantly, than those on the departmental questionnaire. This relationship was consistent for each period as well as for all periods combined. The only other pair of matched items that showed a significant difference was comparison 5, "guidance for study of literature" versus "recommended clerkship readings." Mean values on the institutional questionnaire were lower than the departmental questionnaire for each period and for all periods combined. The relationship was significantly different for the means reflecting periods 2 and 4, and for the mean reflecting all time periods combined. A content analysis of students' comments on the openended questionnaire items indicated that students tended to use specific names in discussing faculty or residents
Comparison of Corresponding Institutional (I) Versus Departmental (D) Questionnaire items for All Six Clerkship Rotations Combined Comparison
n
Mean
t
Residents (I) Residents (D)
142 117
3.47 3.44
0.21630
Independent study time {I) Outside reading time (D)
144 118
2.43 2.61
1.0587
Goals clearly explained (I) Clerkship orientation (D)
142 120
3.52 3.23
2.5411 *
Faculty as teachers (I) Faculty teaching rounds (D)
142 114
3.64 3.31
0.1587
Guidance for study of literature (I) Recommended clerkship readings (D)
144 109
2.95 3.36
3.3903*
* p < 0.01. t = test of significance.
with similar frequency on both questionnaires. The most frequently cited positive aspects of the clerkship for both questionnaires included the patient interaction, the opportunity to learn surgical skills, and the quality of some of the departmental staff. The most frequently cited negative aspects included the lack of time for independent study and the condescending attitude toward students by some of the hospital ward staff. Students did not issue any more calls for policy change on the departmental questionnaires than on the institutional questionnaires. There did seem to be a marked difference between the students' attitudes towards the question on the institutional questionnaire: "If you checked 'Scut Work' please describe what you mean by the term." Many described the nature of scut work but then added that they were not complaining, since this type of work had to be done, recognizing the perspective of the person who may read the questionnaire. COMMENTS A comprehensive assessment of clinical clerkships is of concern to individual faculty members, clinical departments, and the medical school administration. However, a concern that students' perceptions of the clerkship might vary depending on who mandated the evaluation, the time period during which the clerkship was elected, and types of activities required during the clerkship prompted this investigation. The results of this study provide valuable information regarding students' perceptions of the quality of a clerkship relative to their perceptions of the skills and knowledge obtained from clerkship participation. Students appear to quickly reach a plateau in what they perceive to be improvement in their communication skills over the course of an academic year. Since students participate in a variety of clerkships during their third year, they have ample opportunity, regardless of specialty, to hone their communication skills. Therefore, students who participated in the surgery clerkship later in the year did not see as much improvement in communication skills as did students who participated early in the academic year.
THE AMERICAN JOURNAL OF SURGERY
VOLUME 156
SEPTEMBER 1988
157
TEN HAKEN ET AL
Despite the plateau effect in communication skills, students believed their medical skills continued to increase throughout the year. When students thought their medical skills had improved as a result of the clerkship, the clerkship was considered of high quality. Restated, when students were allowed the opportunity to act as a physician, that is, obtain patients' histories and perform physical examinations, construct a differential diagnosis, and outline and participate in a therapeutic plan, they assigned more value to the clerkship experience and hence, rated the clerkship higher than students who experienced less actual patient contact. Students did not alter their responses to clerkship evaluation questionnaires based on whether the instrument was institutionally or departmentally sponsored. Our original bias that students might respond more favorably to a departmentally administered questionnaire proved unfounded. In fact, the medical school assessment consistently rated "faculty as teachers" and "clarity of clerkship goals" higher than did the departmental evaluation. The two comparisons which showed significant differences, "guidance for study of the literature" with "recommended clerkship readings" and "goals clearly explained" with "clerkship orientation," may have resuited from the lack of equivalence between the two pairs. The departmental questionnaire asked very specifically about the readings used to supplement the clerkship, whereas the institutional questionnaire asked about behavior on the part of the instructors to guide students in their literature study. Likewise, the institutional questionnaire addressed whether the goals of the clerkship had been explained to students, whereas the departmental questionnaire asked a global question about the orientation, which could encompass logistics, timing, and so on.
The comments that students included as responses to the open-ended items on both questionnaires addressed similar issues and listed the same complaints regarding the clerkship. Students were equally candid on both questionnaires, indicating lack Of concern for how the information would eventually be used either departmentally or institutionally. There was a significant relationship between perceived improvement of skills required for direct patient contact and students' ratings of the quality of the clerkship. Moreover, previous research has documented a strong correlation between surgery students' perceptions of their own improvement in medical skills and experts' opinions of improvement in these same skills [6]. Thus, this perception of improvement in medical skills on the part of the students may indeed reflect an actual improve-
158
THE AMERICAN JOURNAL O F S U R G E R Y
ment in medical skills. Students clearly defined their personal objectives for clerkship participation as including patient care activities. This study showed that if those objectives are met, the students believe their time has been well spent. The converse is equally true. Students do not think their time is well spent if their clerkship experiences do not include meaningful patient care activities. Clerkship coordinators must be prepared to delineate for students the total value of all clerkship experiences, as well as specifically outline the full range of clerkship objectives. Students who are unaware of these objectives or who are unwilling to accept them as legitimate will continue to underrate the value of the clerkship as a whole. Surgical educators and medical school administrators alike should be heartened by the findings of this study. That assessing clerkships and value assigned to a given experience can be appropriately assigned by either departmental or institutional groups is encouraging but contrary to our preconceived biases. Further, that student perception of the value of a given experience depends in large part on the further development of clinical skills and is not correlated with further development of social and communication skills is important. Finally, that the time of year that the clerkship is elected affects neither student performance nor perception of the clerkship's inherent value is of importance to course directors and suggests that major changes in clerkship design and educational philosophy need not be dictated by the academic calendar, REFERENCES 1. Klintbert IG, Givner N. The comparability of clerkship experiences in a variety of locations. J Med Educ 1977; 52: 518-20. 2. Miller JM, Smith IK, Sosnowski JR, Hester LL. Evaluation of student performance in an obstetrics and gynecology clerkship. J Reprod Med 1982; 27: 443-6. 3, LaPalio LR, Filling CM, Engel JD, Ways PO. Multiple strategies for studying medical clerkship experiences: a case study. J Med Educ 1983; 58: 328-34. 4. Arnold L, Willoughby TL, Calkins EV, Gammon L, Eberhart G. Use of peer evaluation in the assessment of medical students. J Med Educ 1981; 56: 35-42. 5. Morton JB, McBeth WAAG. Correlations between staff, peer and self-assessments of fourth-year students in surgery. J Med Educ 1977; 11: 167-70. 6. Arnold L, Willoughby TL, Calkins EV. Self-evaluation in undergraduate medical education: a longitudinal perspective. J Med Educ 1985; 60: 21-8. 7. Henbest RS, Fersen GS. Preliminary study at the Medical University of Southern Africa on student self-assessment as a means of evaluation. J Med Educ 1985; 60: 66-7. 8. Kegel-Flom P. Predicting supervisor, peer and self ratings of intern performance. J Med Educ 1975; 50: 812-5.
VOLUME 156 SEPTEMBER 1988