- Email: [email protected]
Evaluating the effectiveness of the lecture versus independent study
Evaluation and Program Planning, Vol. 14, pp. 141-146, Printed
1991 Copyright
in the USA. All rights reserved.
0
0149-7189/91 $3.00 + .Oo 1991 Pergamon Press plc
EVALUATING THE EFFECTIVENESS OF THE LECTURE VERSUS INDEPENDENT STUDY DEBRA A. DAROSA and PAUL KOLM Southern
Illinois University
School
of Medicine
HUGH C. FOLLMER University
of Nevada
(L.V.)
L. BEATY PEMBERTON University
of Missouri
(K.C.)
WILLIAMH. PEARCE University
of Colorado
STEPHENLEAPMAN Indiana
University
ABSTRACT The lecture method continues to be the predominant teaching medium in medical education despite widespread emphasis to reduce the lecture hours and make more time available for independent study. This study evaluated the impact on student test scores and study time when taught using the independent study versus lecture approach. Third-year surgery clerks (n = 205) from five medical schools participated in the study. Learning objectives, multiple choice and essay questions were developed for a given topic in surgery. A lecture outline with audiovisuals and an independent study guide were also developed. The teaching method to be used with students in the first rotation was determined randomly for each institution. The alternate teaching method was to be employed in the second rotation. A Lecturer Evaluation Form was used to assess the individuals giving the lecture to measure comparability among lecturers. A Study Habit Survey was completed by students at the end of each clerkship rotation to determine the amount of time and resources used. The written examinations were administered at the end of each clerkship. A two-way analysis of variance was used to determine school and method differences on exam performance, lecturer assessment by students, and self-reported study hours. T tests were used to study differences in study hours between the two groups. The results showed no differences ( p > .OS) in students ’ knowledge as measured by test performance or in the amount of time spent studying. Findings provide added support to medical faculty advocating a decrease in lecture hours and an increase in more active learning strategies. This study was funded in part by the Association for Surgical Education. Requests for reprints should be sent to Debra A. DaRosa, Department P.O. Box 19230, Springfield, IL 62794-9230. 141
of Surgery,
Southern
Illinois
University
School
of Medicine,
142
EVALUATING
DEBRA
THE EFFECTIVENESS
A. DaROSA et al.
OF THE LECTURE
The lecture method continues to be the predominant teaching medium in medical education despite students’ protests of being “lectured to death” and the current emphasis on the merits of active learning (Schwenk & Whitman, 1987). The Association of American Medical Colleges’ report entitled The General Professional Education of the Physician highlighted the need to reduce the number of lecture hours to make more time available for independent study (Muller, 1984). Surgeons attending a recent Association for Surgical Education meeting stated that faculty have been slow to
VERSUS INDEPENDENT
STUDY
implement this recommendation for several reasons including their familiarity with the traditional lecture method, and skepticism about the efficiency of the independent learning approach in terms of time and learning. The purpose of the research described here was to address these concerns by evaluating the two methods by determining if there is a decrease in student knowledge as traditionally measured and an increase in student study hours when lectures are replaced with independent study.
REVIEW OF THE LITERATURE Research investigating the use of the lecture method has shown conflicting results. Some studies find other teaching methods result in better learning outcomes than the lecture approach, while others claim there is not measurable difference between various teaching methods of college instruction (Costin, 1972; Dubin & Taveggio, 1968). Lumsdaine (1963) highlighted flaws in the research methods used to assess the effectiveness of teaching methods claiming insufficient attention was given to the quality of the teaching methods studied. A review by Irby, DeMers, Scher, and Matthew (1976) cited six relevant publications that identified criteria for an “effective lecture.” One of these citations included McKeachie’s (1967) research who found that the use of special techniques enhanced a lecture. Examples of these techniques include summarizing important points at the conclusion of the presentation and using audiovisuals to break up the lecture presentation. He also identified several concepts directly related to a “good” lecture, such as giving examples of a principle and varying context, helping the student identify common elements, giving student feedback, and actively involving the students. Other research indicated that the ideal lecture should start with a meaningful problem posed in such a way that students work to solve it (Berlyne, 1980). Rosenshine and Furst (1973) correlated five variables with lecture effectiveness, including clarity (of organization, explanations, presentation), variability, enthusiasm, task-oriented and/or businesslike behavior, and opportunity for students to learn. Hovland, Janis, and Kelley (1963) emphasized orderliness of presentation as critical to an effective lecture. Irby’s et al. (1976) review of the literature conclude that a “good” lecture format should be based on defined learner needs, provide a strong beginning with objectives and/or lecture outline, be well organized, be varied, explain both sides
of an issue, provide a summary of important points periodically, and achieve closure. Schwenk and Whitman (1984) and Erickson (1985) both highlight the need for preparation, not so much in practicing and rehearsing one’s style of presentation, as in marking out the substance of the talk, its pacing, and the sequence of points. The use of the independent study method has also been scrutinized. Stritter, Buford, Johnson, and Talbert (1973) investigated the effectiveness of self-instructional materials compared to other instructional approaches using scores from the National Board of Medical Examiner’s test as the overall outcome measure. The selfinstructional materials studied by Stritter et al. were designed to allow each student to study individually, proceeding at his or her own learning pace, and had instructional objectives clearly stated in behavioral terms. The use of self-instructional materials resuIted in significantly higher student performance than did other traditional teaching methods. The published research found to date studied the effectiveness of one teaching method by comparing an “experimental” teaching method with one already in use (the “control” method) at the institution where the study was conducted. Lumsdaine (1963) criticized such studies because the teaching methods studied did not necessarily reflect high quality examples. He suggested that for sound conclusions investigators need to take into account the specific characteristics of both instructional methods being studied because in his words “a good film will always beat a poor lecture and vice versa.” Agreeing with Lumsdaine’s observations, the teaching methods for this study were developed to include those characteristics described in the literature that are empirically associated with effective lectures and independent study materials. This was done to ensure a fair comparison of teaching methods.
METHODS Two hundred five third-year medical students from five medical schools who were progressing through one of
two consecutive surgery clerkship rotations participated in the study. The five collaborating medical schools in-
Lecture Versus Independent
eluded Indiana University, Southern Illinois University, University of Colorado, University of Nevada (Las Vegas), and the University of Missouri (Kansas City). The teaching method (lecture or independent study) to be used in the first rotation was determined randomly for each institution. The alternate teaching method was to be employed in the second rotation. Schools indicated the dates of scheduled rotations for the academic year and comparable dates among them were used as starting points for the study to insure that students were at similar stages in the educational process. A posttest-only design was used. This design was selected because subject randomization and/or subject pairing was not feasible because of varying clerkship organizational logistics and the high interaction levels among students within clerkship rotations, which would likely bias the results. Collaborators chose the topic extrahepatic biliary disease as the topic to be taught as all were familiar with it and had some control over its teaching. Learning objectives for the subject were extracted from the objectives written on biliary disease published in the Manual of Surgical Objectives (Lawrence, 1983) which lists objectives for surgery clerkship students. A teaching and evaluation package on extrahepatic biliary disease was developed and sent to all collaborators for review. The package included a detailed lecture outline complete with instructions to the lecturer and slides to be used as an audiovisual aid in the lecture, an Independent Study Guide for students, Lecturer Evaluation Forms, Study Habits Surveys, and multiple-choice and essay examinations. The lecture outline detailed and organized the factual information to be presented during the lecture. The outline listed the learning objectives that served as part of the introduction to the lecture and detailed the progression of information as it was to be presented with examples indicating where during the lecture each slide should be shown and when to stop to summarize throughout the presentation. A summary of the content material highlighting key points was included for presentation at the lecture’s conclusion. The Lecturer Evaluation Form was designed to assess the individuals giving the lectures. It was to be completed anonymously by students following the conclusion of the lecture and collected in a manner that would protect student anonymity. The purpose of this form was to measure comparability among individual lecturers at the various institutions as it was not logistically feasible to have the same person lecture at all participating schools. The Lecturer Evaluation Form used a fivepoint Likert-type rating scale. Lecturers were evaluated in areas related to their delivery (i.e., enthusiasm, clarity of voice, and tonal modulation) as well as organization and quality of presentation. The items on the form were considered to have content validity because they
Study
143
were based on criteria associated with a “good lecture” as identified in the literature. The Independent Study Guide listed the same learning objectives as were communicated in the lecture as well as the references where information concerning each objective could be found. The listed references were the same resources used to outline the lecture. The picture slides shown during the lecture came directly from the reference books; therefore, the content of the lecture and the slides directly corresponded with the reading references listed on the Independent Study Guide. The lecture and independent study guide were also similar in organization of content as the information followed the same sequence of learning objectives. The Independent Study Guide was accompanied by a standard cover letter signed by each respective clerkship director. The cover letter emphasized the importance of the independent study assignment by clearly stating that students were responsible for learning the information designated in the objectives as part of the clerkship requirements. The Study Habit Survey was given to all students for completion in the final few days of each of the two surgery clerkship rotations. A standard introductory paragraph to the form signed by each respective clerkship director emphasized the importance of the information requested. It explained that its purpose was to determine the amount of time and resources used to study basic topics (only one of which was extrahepatic biliary disease) taught during the surgery clerkship. The surveys were to be completed anonymously. A 20-item multiple-choice examination and three open-ended essay questions were developed or selected based on the learning objectives. The multiple-choice questions used in this study were based on questions published by the Association for Surgical Education. The multiple-choice and essay questions were scrutinized by each of the five surgeon collaborators. The multiple-choice questions tested factual recall and comprehension as defined by Bloom (1956). The three essay questions were based on three of the learning objectives and were developed to test at Bloom’s application/synthesis cognitive operation level. Two institutions were unable to administer the essay examinations to their students because it would have been perceived as “unusual.” It was felt that introducing an essay test would overemphasize the importance of biliary disease and the word would trickle to the next rotation of students, thus likely biasing the results. The remaining three schools did not feel the essay examination would be perceived as unusual for it could be incorporated into their traditional examinations which had included open questions in the past. The answer key to the multiple-choice examination was given to the collaborators so each could score their students’ exams. The scores and students’ essay exam-
144
DEBRA A. DaROSA et al.
inations were forwarded to the principle investigator for analysis. Essay questions were sent to one collaborator, who blindly scored them using a prepared answer key. Essays were graded using the traditional O-100% scale. A schedule was forwarded to each collaborator indicating the timetable to be followed for the study. The schedule identified which of the two instructional methods was to be employed in the first rotation of the clerkship (the alternate method to be used in the second rotation) and the weeks in which the instruction and examinations were to be administered. This was done to ensure that all collaborators followed similarly the research protocol to ensure valid comparability of results. A two-way School x Method analysis of variance (ANOVA) was used to determine school and method differences on examination performance, lecture assessment by students and self-reported study hours.
Although method differences (lecture vs. independent study) were of primary interest, the School x Method interaction was important to determine if any differences between the two methods might be peculiar to individual schools. A significant School x Method interaction could indicate differences in application of the two methods at different schools. Depending on the outcome of the overall ANOVA, tests of simple main effects were used to determine differences between methods at each school and/or differences between schools for each method. Since two schools did not administer the essay exam to all students, the ANOVA for the essay exam included three schools rather than five. T tests were used to study differences in study hours between students in the lecture or independent study groups.
RESULTS Two hundred five students from five medical schools participated in the study with 95 taught via the lecture and 110 taught using the independent study method. There were no significant (p > .05) differences between examination scores on the multiple-choice or essay tests between the lectured and independent study groups. Table 1 presents the means and standard deviations of the multiple-choice and essay exams for lecture and independent study groups. The ANOVA for the multiple-choice exam resulted in a significant main effect for School (F(4, 195) = 9.60, p = .OOOl), a nonsignificant main effect for Method (F(l, 195) = 2.26, p = .1342), and a nonsignificant School x Method interaction (F(4, 195) = 1.99, p = .0972). The significant School main effect resulted from the higher mean multiple choice examination score for School A for both the lecture and the independent study method. On the average, students in the lecture and independent study groups scored 12.8 and 12.2, respectively, out of 20 on the multiple-choice questions. The essay questions were administered by three of the collaborating schools to students in both groups; one school gave the essay questions only to the lecture group due to an unanticipated logistical problem. Forty-three lectured students and 59 independent study group students completed the essay questions. The ANOVA resulted in a significant School main effect (F(2, 91) = 9.28, p = .0002), a nonsignificant Method main effect (F(l, 91) = .09, p = .7606) and a nonsignificant School x Method interaction (F(2, 91) = .48, p = .6195). The significant School main effect resulted from the higher mean scores of School C for both the lecture and independent study groups (Table 1). For each school, there was no significant difference between lecture method and independent study method scores. On the average, students in the lecture group scored
63.1 on the essay exams, and students in the independent group scored 58.1 out of 100 possible points. Fifty-eight students (55%) attending the lecture on extrahepatic biliary disease at four medical schools completed the Lecturer Evaluation Form. The mean ratings at three medical schools reflected high overall ratings ranging from 4.3 to 4.6 on a one-to-five Likerttype scale with one indicating poor and five representing excellent. One school received poor ratings from students with an average overall rating of 2.6. Results of the analysis of variance showed no significant differences between lecturer ratings at three schools but there was a significant difference (p < .05) in ratings at the school receiving the lower ratings. Study Survey Forms were completed by 72 students with 49 from the lecture and 23 from the independent study groups. Students taught via lecture method estimated spending 6.0 h and students involved in indepen-
TABLE 1 MEANS AND STANDARD DEVIATIONS OF EXAMINATION SCORES REPORTED BY SCHOOL AND TEACHING METHOD Multiple Choice Exam (20 possible points)
Lecture
(n= 95)
Independent Study (n= 110)
School
M
(SD)
M
(SD)
A 0 C D E
14.7 13.2 12.2 12.2 12.5
(1.4) (0.8) (1.6) (1.9) (1.3)
13.0 12.8 11.7 12.2 12.9
(2.0) (1.6) (1.4) (1.3) (1.6)
Essay Exam (100 possible points)
Lecture (n= 43) M
(SD)
53.3 (8.3) 62.2 (10.5) 64.9 (8.4) No data 52.5 (21.)
Independent Study (n = 59) M 47.0 No 63.8 No 56.7
(SD) (11.1) data (10.9) data (18.8)
Lecture Versus Independent dent study estimated 4.9 h. These estimates included all time spent learning about extrahepatic biliary disease ineluding reading, reviewing, and lecture or independent
Study
145
study time. The results from a t-test analysis showed no significant (t(70) = 1.15, p > .05) difference between students in the lecture and independent study groups.
DISCUSSION The results of this study showed four out of five schools found no differences as measured by test performance when students were taught using a lecture compared to independent study approach. One medical school did find a difference on multiple-choice performance with students attending the lecture scoring higher than those in the independent study program. This difference may have occurred at this school, as opposed to the others, because the lecture method is used very infrequently, and therefore, students may have sensed an accentuation on the topic and paid more attention than they would have normally. Study time, defined here as the number of hours spent learning about extrahepatic biliary disease, was not different as reported by students taught via the lecture and the independent method approach. The comparability of study hours for the two groups has efficiency implications. Students may not be comfortable relying solely on hearing a lecture for learning about a subject as they spend a significant number of hours studying the topic on which they were lectured. This finding addresses popular criticisms that the independent or self-instructional approach is too inefficient of a method requiring too many study hours. This research demonstrates that guided independent study is at least as time efficient as a lecture for the learner, and does not require one hour of faculty time to present. One school had a poor lecturer as rated by students, but this did not effect examination scores. The specific teaching behaviors receiving the lowest rating as assessed by students at that institution included enthusiasm and clarity of voice. The unaffected test scores may indicate that the students’ knowledge of the learning objectives, communicated at the start of the lecture, may have been all students needed to know to pass examinations. Another explanation may be that students are disciplined to listen attentively for content to faculty lectures regardless of the lecturer’s ability to present well. A collaborative research study at five medical schools differing geographically posed some logistical problems. The response rate of the Study Survey was good at those schools where the students remained in the same hospital for the following clinical rotation. Although students were asked to return these surveys in the mail anonymously, follow-up at two institutions where students changed locations was poor resulting in an overall response rate of 35%. Additionally, although the majority of students in the lecture group completed the Lecture Evaluation Form, 45% of them did not. This
may have occurred because of different emphases by collaborators given to completing the form and diligence employed in collecting them. The results of these two surveys should be interpreted in light of their response rates. Besides these deviations, however, the methods and data collection efforts used at the five schools were uniform. Each used standardized learning objectives, lecture outlines with slides, independent study guide sheets for students with uniform instructions, and cover letter, examination questions, scoring keys, rating forms, and study habit surveys. The instruction, test administration, and survey distribution were all given at comparable periods during the various clerkship rotations. Research findings reported here are based on a comparison of two teaching methods employed at medical schools involving third-year students rotating on a surgery clerkship. The fact that medical students served as the sample needs to be taken into consideration when interpreting results. Medical students are highly motivated individuals who, in the research reported here, had completed 4 years of college and 2 years of medical school with the latter predominantly involving large class lectures. Therefore, the results are more likely generalizable to mature adult learners who have acquired the majority of their knowledge through reading and didactic teaching methods. Additionally, this study focused on comparing the results associated with two teaching methods covering one topic. Influencing variables such as topic difficulty, appeal, and perceptions of significance to the learners could affect the outcome. The GPEP report emphasized the need to reduce lecture hours and encourage, as well as provide time for, more independent study. Collaborators from each of the five schools involved in this study felt faculty were reluctant to reduce lecture hours largely because it was an efficient teaching approach and because of concerns that test performance would suffer. This evaluation research effort provides supportive evidence that student examination scores would not likely be sacrificed if a lecture was replaced with a structured independent study approach. Although these results suggest that test performance and study time will not be compromised using a nonlecture teaching method, the effects on learning and application capability require further study. Multiple-choice and essay tests, although the most widely used testing methods in medical education, do not necessarily yield scores reflective of actual learning. Additional research examining the effectiveness of independent study is
146
DEBRA A. DaROSA et al.
needed to explore learning and retention issues beyond traditional examination performance. The study described here provides a model for future investigation but some modifications would strengthen the findings. Conducting a training or “polishing” session for lecturers to ensure more homogeneity would be helpful as would involving a nonpa~is~ education specialist to evaluate each lecture. The data could be used as a crossreference with the student evaluation of lectures. These evaluation forms, as well as the other self-report survey (Study Habit Survey) should be more closely followed
in order to achieve a higher response rate. Finally, it might be of interest to investigate which method (lecture or independent study) is more desirable to students. Students have different learning styles which may correlate more closely with one or the other method. Resuits of such studies could provide further evidence for convincing lecture proponent faculty to implement GPEP recommendations to reduce the number of lectures and increase time available for independent or self-directed active learning opportunities.
REFERENCES BERLYNE, D.E. (1960). Conji’ict, arousal, and conflict. New York: McGraw Hill. BLOOM, B.S. (Ed.). (1956). Taxonomy of educutjona~ objeciives, handbook I: The cognitive domain. New York: McKay. COSTIN, R. (1972). Lecturing versus other methods of teaching: A review of research. British Journal of Educational Technofogy, 9, 36-47. DUBIN, R., & TAVEGGIO, T.C. (1968). The teaching learningparadox. Eugene, OR: Center for the Advanced Study of Educational Administration.
LUMSDAINE, A.A. (1963). Instruments and media of instruction. in N.L. Gage (Ed.), Handbook of research and teaching (pp. 583682). Chicago: Rand McNally. McKEACHIE, W.J. (1967). Research in teaching: The gap between theory and practice. In C.B. Lee (Ed.), rmproving rorlege teaching (pp. 21 l-239). Washington, DC: American Council on Education. MULLER, S. (1984). Physicians for the twenty-first century: Report of the project panel on the general professional education of the physician and college preparation for medicine. Journal of Medical Education, 59, l-208.
ERICKSEN, S.C. (1985). The essence of good teaching. San Francisco: Jossey-Bass.
ROSENSHINE, B., & FURST, N. (1973). The use of direct observation to study teaching. In R.M.W. Travers (Ed.), Second handbook of research on teaching. Chicago: Rand McNafIy.
HOVLAND, CL, JANIS, I.L., & KELLEY, H.H. (1963). Communication andpersuasion. New Haven, CT: Yale University Press.
SCHWENK, T.L., & WHITMAN, N. (1984). Residents as teachers: A guide to educationalpractice. Salt Lake City: University of Utah.
IRBY, D., DEMERS, J., SCHER, M., & MATTHEW, D. (1976). A model for the improvement of medical faculty lecturing. Journal of Medical Education, 51, 403-409.
SCHWENK, T.L., & WHITMAN, N. (1987). The physician as teacher. Baltimore: Williams & Wilkens.
LAWRENCE, P. (Ed.). Manual of surgicaI objectives. Springfield, IL: Association for Surgical Education.
STRITTER, F.R., BUFORD, H.J., JOHNSON, S-R., & TALBERT, L.M. (1973). Documentation of the effectiveness of self-instructional materials. JOUF~ZQ! of Medicai Education, 48, 1129-l 132.
1991 Copyright
in the USA. All rights reserved.
0
0149-7189/91 $3.00 + .Oo 1991 Pergamon Press plc
EVALUATING THE EFFECTIVENESS OF THE LECTURE VERSUS INDEPENDENT STUDY DEBRA A. DAROSA and PAUL KOLM Southern
Illinois University
School
of Medicine
HUGH C. FOLLMER University
of Nevada
(L.V.)
L. BEATY PEMBERTON University
of Missouri
(K.C.)
WILLIAMH. PEARCE University
of Colorado
STEPHENLEAPMAN Indiana
University
ABSTRACT The lecture method continues to be the predominant teaching medium in medical education despite widespread emphasis to reduce the lecture hours and make more time available for independent study. This study evaluated the impact on student test scores and study time when taught using the independent study versus lecture approach. Third-year surgery clerks (n = 205) from five medical schools participated in the study. Learning objectives, multiple choice and essay questions were developed for a given topic in surgery. A lecture outline with audiovisuals and an independent study guide were also developed. The teaching method to be used with students in the first rotation was determined randomly for each institution. The alternate teaching method was to be employed in the second rotation. A Lecturer Evaluation Form was used to assess the individuals giving the lecture to measure comparability among lecturers. A Study Habit Survey was completed by students at the end of each clerkship rotation to determine the amount of time and resources used. The written examinations were administered at the end of each clerkship. A two-way analysis of variance was used to determine school and method differences on exam performance, lecturer assessment by students, and self-reported study hours. T tests were used to study differences in study hours between the two groups. The results showed no differences ( p > .OS) in students ’ knowledge as measured by test performance or in the amount of time spent studying. Findings provide added support to medical faculty advocating a decrease in lecture hours and an increase in more active learning strategies. This study was funded in part by the Association for Surgical Education. Requests for reprints should be sent to Debra A. DaRosa, Department P.O. Box 19230, Springfield, IL 62794-9230. 141
of Surgery,
Southern
Illinois
University
School
of Medicine,
142
EVALUATING
DEBRA
THE EFFECTIVENESS
A. DaROSA et al.
OF THE LECTURE
The lecture method continues to be the predominant teaching medium in medical education despite students’ protests of being “lectured to death” and the current emphasis on the merits of active learning (Schwenk & Whitman, 1987). The Association of American Medical Colleges’ report entitled The General Professional Education of the Physician highlighted the need to reduce the number of lecture hours to make more time available for independent study (Muller, 1984). Surgeons attending a recent Association for Surgical Education meeting stated that faculty have been slow to
VERSUS INDEPENDENT
STUDY
implement this recommendation for several reasons including their familiarity with the traditional lecture method, and skepticism about the efficiency of the independent learning approach in terms of time and learning. The purpose of the research described here was to address these concerns by evaluating the two methods by determining if there is a decrease in student knowledge as traditionally measured and an increase in student study hours when lectures are replaced with independent study.
REVIEW OF THE LITERATURE Research investigating the use of the lecture method has shown conflicting results. Some studies find other teaching methods result in better learning outcomes than the lecture approach, while others claim there is not measurable difference between various teaching methods of college instruction (Costin, 1972; Dubin & Taveggio, 1968). Lumsdaine (1963) highlighted flaws in the research methods used to assess the effectiveness of teaching methods claiming insufficient attention was given to the quality of the teaching methods studied. A review by Irby, DeMers, Scher, and Matthew (1976) cited six relevant publications that identified criteria for an “effective lecture.” One of these citations included McKeachie’s (1967) research who found that the use of special techniques enhanced a lecture. Examples of these techniques include summarizing important points at the conclusion of the presentation and using audiovisuals to break up the lecture presentation. He also identified several concepts directly related to a “good” lecture, such as giving examples of a principle and varying context, helping the student identify common elements, giving student feedback, and actively involving the students. Other research indicated that the ideal lecture should start with a meaningful problem posed in such a way that students work to solve it (Berlyne, 1980). Rosenshine and Furst (1973) correlated five variables with lecture effectiveness, including clarity (of organization, explanations, presentation), variability, enthusiasm, task-oriented and/or businesslike behavior, and opportunity for students to learn. Hovland, Janis, and Kelley (1963) emphasized orderliness of presentation as critical to an effective lecture. Irby’s et al. (1976) review of the literature conclude that a “good” lecture format should be based on defined learner needs, provide a strong beginning with objectives and/or lecture outline, be well organized, be varied, explain both sides
of an issue, provide a summary of important points periodically, and achieve closure. Schwenk and Whitman (1984) and Erickson (1985) both highlight the need for preparation, not so much in practicing and rehearsing one’s style of presentation, as in marking out the substance of the talk, its pacing, and the sequence of points. The use of the independent study method has also been scrutinized. Stritter, Buford, Johnson, and Talbert (1973) investigated the effectiveness of self-instructional materials compared to other instructional approaches using scores from the National Board of Medical Examiner’s test as the overall outcome measure. The selfinstructional materials studied by Stritter et al. were designed to allow each student to study individually, proceeding at his or her own learning pace, and had instructional objectives clearly stated in behavioral terms. The use of self-instructional materials resuIted in significantly higher student performance than did other traditional teaching methods. The published research found to date studied the effectiveness of one teaching method by comparing an “experimental” teaching method with one already in use (the “control” method) at the institution where the study was conducted. Lumsdaine (1963) criticized such studies because the teaching methods studied did not necessarily reflect high quality examples. He suggested that for sound conclusions investigators need to take into account the specific characteristics of both instructional methods being studied because in his words “a good film will always beat a poor lecture and vice versa.” Agreeing with Lumsdaine’s observations, the teaching methods for this study were developed to include those characteristics described in the literature that are empirically associated with effective lectures and independent study materials. This was done to ensure a fair comparison of teaching methods.
METHODS Two hundred five third-year medical students from five medical schools who were progressing through one of
two consecutive surgery clerkship rotations participated in the study. The five collaborating medical schools in-
Lecture Versus Independent
eluded Indiana University, Southern Illinois University, University of Colorado, University of Nevada (Las Vegas), and the University of Missouri (Kansas City). The teaching method (lecture or independent study) to be used in the first rotation was determined randomly for each institution. The alternate teaching method was to be employed in the second rotation. Schools indicated the dates of scheduled rotations for the academic year and comparable dates among them were used as starting points for the study to insure that students were at similar stages in the educational process. A posttest-only design was used. This design was selected because subject randomization and/or subject pairing was not feasible because of varying clerkship organizational logistics and the high interaction levels among students within clerkship rotations, which would likely bias the results. Collaborators chose the topic extrahepatic biliary disease as the topic to be taught as all were familiar with it and had some control over its teaching. Learning objectives for the subject were extracted from the objectives written on biliary disease published in the Manual of Surgical Objectives (Lawrence, 1983) which lists objectives for surgery clerkship students. A teaching and evaluation package on extrahepatic biliary disease was developed and sent to all collaborators for review. The package included a detailed lecture outline complete with instructions to the lecturer and slides to be used as an audiovisual aid in the lecture, an Independent Study Guide for students, Lecturer Evaluation Forms, Study Habits Surveys, and multiple-choice and essay examinations. The lecture outline detailed and organized the factual information to be presented during the lecture. The outline listed the learning objectives that served as part of the introduction to the lecture and detailed the progression of information as it was to be presented with examples indicating where during the lecture each slide should be shown and when to stop to summarize throughout the presentation. A summary of the content material highlighting key points was included for presentation at the lecture’s conclusion. The Lecturer Evaluation Form was designed to assess the individuals giving the lectures. It was to be completed anonymously by students following the conclusion of the lecture and collected in a manner that would protect student anonymity. The purpose of this form was to measure comparability among individual lecturers at the various institutions as it was not logistically feasible to have the same person lecture at all participating schools. The Lecturer Evaluation Form used a fivepoint Likert-type rating scale. Lecturers were evaluated in areas related to their delivery (i.e., enthusiasm, clarity of voice, and tonal modulation) as well as organization and quality of presentation. The items on the form were considered to have content validity because they
Study
143
were based on criteria associated with a “good lecture” as identified in the literature. The Independent Study Guide listed the same learning objectives as were communicated in the lecture as well as the references where information concerning each objective could be found. The listed references were the same resources used to outline the lecture. The picture slides shown during the lecture came directly from the reference books; therefore, the content of the lecture and the slides directly corresponded with the reading references listed on the Independent Study Guide. The lecture and independent study guide were also similar in organization of content as the information followed the same sequence of learning objectives. The Independent Study Guide was accompanied by a standard cover letter signed by each respective clerkship director. The cover letter emphasized the importance of the independent study assignment by clearly stating that students were responsible for learning the information designated in the objectives as part of the clerkship requirements. The Study Habit Survey was given to all students for completion in the final few days of each of the two surgery clerkship rotations. A standard introductory paragraph to the form signed by each respective clerkship director emphasized the importance of the information requested. It explained that its purpose was to determine the amount of time and resources used to study basic topics (only one of which was extrahepatic biliary disease) taught during the surgery clerkship. The surveys were to be completed anonymously. A 20-item multiple-choice examination and three open-ended essay questions were developed or selected based on the learning objectives. The multiple-choice questions used in this study were based on questions published by the Association for Surgical Education. The multiple-choice and essay questions were scrutinized by each of the five surgeon collaborators. The multiple-choice questions tested factual recall and comprehension as defined by Bloom (1956). The three essay questions were based on three of the learning objectives and were developed to test at Bloom’s application/synthesis cognitive operation level. Two institutions were unable to administer the essay examinations to their students because it would have been perceived as “unusual.” It was felt that introducing an essay test would overemphasize the importance of biliary disease and the word would trickle to the next rotation of students, thus likely biasing the results. The remaining three schools did not feel the essay examination would be perceived as unusual for it could be incorporated into their traditional examinations which had included open questions in the past. The answer key to the multiple-choice examination was given to the collaborators so each could score their students’ exams. The scores and students’ essay exam-
144
DEBRA A. DaROSA et al.
inations were forwarded to the principle investigator for analysis. Essay questions were sent to one collaborator, who blindly scored them using a prepared answer key. Essays were graded using the traditional O-100% scale. A schedule was forwarded to each collaborator indicating the timetable to be followed for the study. The schedule identified which of the two instructional methods was to be employed in the first rotation of the clerkship (the alternate method to be used in the second rotation) and the weeks in which the instruction and examinations were to be administered. This was done to ensure that all collaborators followed similarly the research protocol to ensure valid comparability of results. A two-way School x Method analysis of variance (ANOVA) was used to determine school and method differences on examination performance, lecture assessment by students and self-reported study hours.
Although method differences (lecture vs. independent study) were of primary interest, the School x Method interaction was important to determine if any differences between the two methods might be peculiar to individual schools. A significant School x Method interaction could indicate differences in application of the two methods at different schools. Depending on the outcome of the overall ANOVA, tests of simple main effects were used to determine differences between methods at each school and/or differences between schools for each method. Since two schools did not administer the essay exam to all students, the ANOVA for the essay exam included three schools rather than five. T tests were used to study differences in study hours between students in the lecture or independent study groups.
RESULTS Two hundred five students from five medical schools participated in the study with 95 taught via the lecture and 110 taught using the independent study method. There were no significant (p > .05) differences between examination scores on the multiple-choice or essay tests between the lectured and independent study groups. Table 1 presents the means and standard deviations of the multiple-choice and essay exams for lecture and independent study groups. The ANOVA for the multiple-choice exam resulted in a significant main effect for School (F(4, 195) = 9.60, p = .OOOl), a nonsignificant main effect for Method (F(l, 195) = 2.26, p = .1342), and a nonsignificant School x Method interaction (F(4, 195) = 1.99, p = .0972). The significant School main effect resulted from the higher mean multiple choice examination score for School A for both the lecture and the independent study method. On the average, students in the lecture and independent study groups scored 12.8 and 12.2, respectively, out of 20 on the multiple-choice questions. The essay questions were administered by three of the collaborating schools to students in both groups; one school gave the essay questions only to the lecture group due to an unanticipated logistical problem. Forty-three lectured students and 59 independent study group students completed the essay questions. The ANOVA resulted in a significant School main effect (F(2, 91) = 9.28, p = .0002), a nonsignificant Method main effect (F(l, 91) = .09, p = .7606) and a nonsignificant School x Method interaction (F(2, 91) = .48, p = .6195). The significant School main effect resulted from the higher mean scores of School C for both the lecture and independent study groups (Table 1). For each school, there was no significant difference between lecture method and independent study method scores. On the average, students in the lecture group scored
63.1 on the essay exams, and students in the independent group scored 58.1 out of 100 possible points. Fifty-eight students (55%) attending the lecture on extrahepatic biliary disease at four medical schools completed the Lecturer Evaluation Form. The mean ratings at three medical schools reflected high overall ratings ranging from 4.3 to 4.6 on a one-to-five Likerttype scale with one indicating poor and five representing excellent. One school received poor ratings from students with an average overall rating of 2.6. Results of the analysis of variance showed no significant differences between lecturer ratings at three schools but there was a significant difference (p < .05) in ratings at the school receiving the lower ratings. Study Survey Forms were completed by 72 students with 49 from the lecture and 23 from the independent study groups. Students taught via lecture method estimated spending 6.0 h and students involved in indepen-
TABLE 1 MEANS AND STANDARD DEVIATIONS OF EXAMINATION SCORES REPORTED BY SCHOOL AND TEACHING METHOD Multiple Choice Exam (20 possible points)
Lecture
(n= 95)
Independent Study (n= 110)
School
M
(SD)
M
(SD)
A 0 C D E
14.7 13.2 12.2 12.2 12.5
(1.4) (0.8) (1.6) (1.9) (1.3)
13.0 12.8 11.7 12.2 12.9
(2.0) (1.6) (1.4) (1.3) (1.6)
Essay Exam (100 possible points)
Lecture (n= 43) M
(SD)
53.3 (8.3) 62.2 (10.5) 64.9 (8.4) No data 52.5 (21.)
Independent Study (n = 59) M 47.0 No 63.8 No 56.7
(SD) (11.1) data (10.9) data (18.8)
Lecture Versus Independent dent study estimated 4.9 h. These estimates included all time spent learning about extrahepatic biliary disease ineluding reading, reviewing, and lecture or independent
Study
145
study time. The results from a t-test analysis showed no significant (t(70) = 1.15, p > .05) difference between students in the lecture and independent study groups.
DISCUSSION The results of this study showed four out of five schools found no differences as measured by test performance when students were taught using a lecture compared to independent study approach. One medical school did find a difference on multiple-choice performance with students attending the lecture scoring higher than those in the independent study program. This difference may have occurred at this school, as opposed to the others, because the lecture method is used very infrequently, and therefore, students may have sensed an accentuation on the topic and paid more attention than they would have normally. Study time, defined here as the number of hours spent learning about extrahepatic biliary disease, was not different as reported by students taught via the lecture and the independent method approach. The comparability of study hours for the two groups has efficiency implications. Students may not be comfortable relying solely on hearing a lecture for learning about a subject as they spend a significant number of hours studying the topic on which they were lectured. This finding addresses popular criticisms that the independent or self-instructional approach is too inefficient of a method requiring too many study hours. This research demonstrates that guided independent study is at least as time efficient as a lecture for the learner, and does not require one hour of faculty time to present. One school had a poor lecturer as rated by students, but this did not effect examination scores. The specific teaching behaviors receiving the lowest rating as assessed by students at that institution included enthusiasm and clarity of voice. The unaffected test scores may indicate that the students’ knowledge of the learning objectives, communicated at the start of the lecture, may have been all students needed to know to pass examinations. Another explanation may be that students are disciplined to listen attentively for content to faculty lectures regardless of the lecturer’s ability to present well. A collaborative research study at five medical schools differing geographically posed some logistical problems. The response rate of the Study Survey was good at those schools where the students remained in the same hospital for the following clinical rotation. Although students were asked to return these surveys in the mail anonymously, follow-up at two institutions where students changed locations was poor resulting in an overall response rate of 35%. Additionally, although the majority of students in the lecture group completed the Lecture Evaluation Form, 45% of them did not. This
may have occurred because of different emphases by collaborators given to completing the form and diligence employed in collecting them. The results of these two surveys should be interpreted in light of their response rates. Besides these deviations, however, the methods and data collection efforts used at the five schools were uniform. Each used standardized learning objectives, lecture outlines with slides, independent study guide sheets for students with uniform instructions, and cover letter, examination questions, scoring keys, rating forms, and study habit surveys. The instruction, test administration, and survey distribution were all given at comparable periods during the various clerkship rotations. Research findings reported here are based on a comparison of two teaching methods employed at medical schools involving third-year students rotating on a surgery clerkship. The fact that medical students served as the sample needs to be taken into consideration when interpreting results. Medical students are highly motivated individuals who, in the research reported here, had completed 4 years of college and 2 years of medical school with the latter predominantly involving large class lectures. Therefore, the results are more likely generalizable to mature adult learners who have acquired the majority of their knowledge through reading and didactic teaching methods. Additionally, this study focused on comparing the results associated with two teaching methods covering one topic. Influencing variables such as topic difficulty, appeal, and perceptions of significance to the learners could affect the outcome. The GPEP report emphasized the need to reduce lecture hours and encourage, as well as provide time for, more independent study. Collaborators from each of the five schools involved in this study felt faculty were reluctant to reduce lecture hours largely because it was an efficient teaching approach and because of concerns that test performance would suffer. This evaluation research effort provides supportive evidence that student examination scores would not likely be sacrificed if a lecture was replaced with a structured independent study approach. Although these results suggest that test performance and study time will not be compromised using a nonlecture teaching method, the effects on learning and application capability require further study. Multiple-choice and essay tests, although the most widely used testing methods in medical education, do not necessarily yield scores reflective of actual learning. Additional research examining the effectiveness of independent study is
146
DEBRA A. DaROSA et al.
needed to explore learning and retention issues beyond traditional examination performance. The study described here provides a model for future investigation but some modifications would strengthen the findings. Conducting a training or “polishing” session for lecturers to ensure more homogeneity would be helpful as would involving a nonpa~is~ education specialist to evaluate each lecture. The data could be used as a crossreference with the student evaluation of lectures. These evaluation forms, as well as the other self-report survey (Study Habit Survey) should be more closely followed
in order to achieve a higher response rate. Finally, it might be of interest to investigate which method (lecture or independent study) is more desirable to students. Students have different learning styles which may correlate more closely with one or the other method. Resuits of such studies could provide further evidence for convincing lecture proponent faculty to implement GPEP recommendations to reduce the number of lectures and increase time available for independent or self-directed active learning opportunities.
REFERENCES BERLYNE, D.E. (1960). Conji’ict, arousal, and conflict. New York: McGraw Hill. BLOOM, B.S. (Ed.). (1956). Taxonomy of educutjona~ objeciives, handbook I: The cognitive domain. New York: McKay. COSTIN, R. (1972). Lecturing versus other methods of teaching: A review of research. British Journal of Educational Technofogy, 9, 36-47. DUBIN, R., & TAVEGGIO, T.C. (1968). The teaching learningparadox. Eugene, OR: Center for the Advanced Study of Educational Administration.
LUMSDAINE, A.A. (1963). Instruments and media of instruction. in N.L. Gage (Ed.), Handbook of research and teaching (pp. 583682). Chicago: Rand McNally. McKEACHIE, W.J. (1967). Research in teaching: The gap between theory and practice. In C.B. Lee (Ed.), rmproving rorlege teaching (pp. 21 l-239). Washington, DC: American Council on Education. MULLER, S. (1984). Physicians for the twenty-first century: Report of the project panel on the general professional education of the physician and college preparation for medicine. Journal of Medical Education, 59, l-208.
ERICKSEN, S.C. (1985). The essence of good teaching. San Francisco: Jossey-Bass.
ROSENSHINE, B., & FURST, N. (1973). The use of direct observation to study teaching. In R.M.W. Travers (Ed.), Second handbook of research on teaching. Chicago: Rand McNafIy.
HOVLAND, CL, JANIS, I.L., & KELLEY, H.H. (1963). Communication andpersuasion. New Haven, CT: Yale University Press.
SCHWENK, T.L., & WHITMAN, N. (1984). Residents as teachers: A guide to educationalpractice. Salt Lake City: University of Utah.
IRBY, D., DEMERS, J., SCHER, M., & MATTHEW, D. (1976). A model for the improvement of medical faculty lecturing. Journal of Medical Education, 51, 403-409.
SCHWENK, T.L., & WHITMAN, N. (1987). The physician as teacher. Baltimore: Williams & Wilkens.
LAWRENCE, P. (Ed.). Manual of surgicaI objectives. Springfield, IL: Association for Surgical Education.
STRITTER, F.R., BUFORD, H.J., JOHNSON, S-R., & TALBERT, L.M. (1973). Documentation of the effectiveness of self-instructional materials. JOUF~ZQ! of Medicai Education, 48, 1129-l 132.