- Email: [email protected]
Ecological validity of adjunct questions in a college biology course
CONTEMPORARY
EDUCATIONAL
Ecological
PSYCHOLOGY
Validity
CARL SPRING, JULIUS
11,
79-89 (1986)
of Adjunct Questions Biology Course
in a College
SASSENRATH,
KETELLAPPER
lini~~rsity
AND HENDRIK
of Ca/iforni~~,
Dn\~is
To validate their use in a natural environment with realistic learning materials and conditions, the efficacy of adjunct questions was tested in a college biology course with a double-crossover design. Mixed results were obtained. On the one hand, students using adjunct questions during the second half of the course scored significantly higher than control students on a variety of measures obtained on the final examination. Significant indirect as well as direct effects were obtained, and significant effects were demonstrated on comprehension as well as verbatim-recall examination questions. On the other hand, during the first half of the course evidence was also obtained that some students did not follow instructions on the proper use of adjunct questions, and that for those students adjunct questions actually impaired learning. Such vagaries may be common in natural experimental settings as opposed to highly controlled laboratory settings. It is concluded that, rather than requiring all students to use adjunct questions. it would be better to make them optional. Presumably, adjunct questions would be used. under an optional condition, only by students who were motivated to use them properly after receiving strong warnings concerning their possible misuse. ‘C IOXh Academc Press. 1°C
A good deal of research has been directed toward the study of instructional procedures which, although promising, never find their way into classrooms. Research on the use of adjunct questions to improve learning is a case in point. Adjunct questions are questions which are inserted at critical points within a text to facilitate a student’s understanding and retention of the material. Starting with the seminal work of Rothkopf (1965), there have been a large number of experimental studies that have investigated the use of adjunct questions. The frequency of these investigations peaked during the last decade and is now in decline. Although most of these studies demonstrated that adjunct questions can facilitate learning (R. C. Anderson & Biddle, 1973, this research has had little influence on educational practice. One possible reason for this gap between research and practice is that instructional studies are often conducted under unnaturally controlled Financial assistance for this research came from the Basic Skills Research Program, University of California, Davis. We thank Bruce Campell and Daryl Koutnik for writing the adjunct questions, and Sam Neustadt for assistance in analyzing the data. Send requests for
reprints to Dr. Julius Sassenrath, Department of Education, University of California. Davis, CA 95616. 19 0361-476X186 $3.00 Copyright ‘c 1986 by Academic Preaa. Inc. All rights of reproduction in any form revxve4
80
SPRING,
SASSENRATH,
AND
KETELLAPPER
conditions and for unrealistically short durations. They are perceived by practitioners, therefore, as having insufficient ecological validity. This is especially true of research on adjunct questions. Although the effects of adjunct questions are well understood under laboratory controlled conditions with short, specially constructed reading passages, their effects with relatively uncontrolled homework reading of a large textbook are not well understood (Faw & Waller, 1976). To correct this problem, the present study was conducted in a college biology course under conditions which differed from typical laboratory studies in several ways that should improve its ecological validity: (I) the duration of the study was much longer, (2) the reading assignments were considerably larger, (3) the material to be learned was more realistic and more conceptually integrated, and (4) it was not possible to monitor the students as they read the textbook and answered adjunct questions. Because our objective was primarily applied rather than theoretical, we used several findings from laboratory studies in an attempt to maximize the probable impact of the experimental treatment. For example, it is known that adjunct questions presented after reading relevant sections of a text are more effective than questions presented before the relevant text, and that this difference is larger for text material not specifically covered by adjunct questions (indirect effect) than for material covered by adjunct questions (direct effect) (R. C. Anderson & Biddle, 1975). Also these authors point out that some studies have shown that reading adjunct questions before reading the text may actually inhibit indirect learning. In the present study, therefore, students in the experimental groups were instructed to read and answer adjunct questions after rather than before reading relevant text material. Also from the review by R. C. Anderson and Biddle (1975) we know that (1) adjunct questions requiring overt written responses are more effective than those requiring covert mental responses, (2) adjunct questions requiring constructed answers are more effective than those requiring multiple-choice answers, and (3) answering adjunct questions after reading one or two paragraphs is more effective than answering them after reading longer sections of text. These results from laboratory studies were also used in designing the experimental treatment so as to maximize its probable impact. In addition to our general objective of maximizing the impact of the experimental treatment, we had several other objectives. First, we wanted to compare the direct and indirect effects of answering adjunct questions. In most laboratory studies the direct effects are considerably larger than the indirect effects, and in some studies the indirect effects have been negligible (R. C. Anderson & Biddle, 1975). We reasoned, however, that with the highly integrated material covered in a biology textbook, answering an adjunct question might improve one’s memory
VALIDITY
OF ADJUNCT
QUESTIONS
81
for concepts that are related to the question as well as for concepts that are actually covered by the question. In this case, we would expect a significant indirect effect as well as a direct effect. Second, we wanted to determine whether students would, as instructed, read adjunct questions after rather than before reading the relevant text material. This was determined with a questionnaire administered at the end of the course. Related to this, we also wanted to measure the direct and indirect effects of failing to follow instructions. Based on laboratory evidence discussed above, it was expected that not following instructions would reduce the indirect effect more than the direct effect. Finally, we wanted to determine whether adjunct questions are as effective in improving comprehension as they are in improving verbatim recall. To determine this, we included both types of questions on the midterm and final examinations from which the effects of the experimental treatment were measured. METHOD
Design and Subjects The one-quarter introductory biology course was for students at the University of California, Davis Campus, not majoring in a biological science. The course was divided into two sessions of equal duration. The first session was followed by a midterm multiple-choice examination on material covered during the first half of the course. The second session was followed by a final multiple-choice examination, a large portion of which covered material from the second half of the course. During the first half of the course, about half of the students were required to complete adjunct questions (experimental group), and the other half were not (control group). During the second half of the course, group assignments were reversed, resulting in an AB, BA design. The primary reason for using the crossover design was to be fair to students in the class by having them experience both the treatment and control conditions. The mean scores of the experimental and control groups were compared on the midterm examination, but only on questions covering material from the textbook. After group assignments were reversed halfway through the course, the mean scores of the new experimental and control groups were again compared on the final examination. For the latter comparison, however, only questions on the final examination which covered material presented during the second half of the course from the textbook were considered. For practical reasons, all students within a discussion section were assigned to the same group, and discussion sections of approximately 20 students each were randomly assigned to the two groups. This resulted in the assignment of 97 students to the group which received the experimental treatment preceding the midterm and 86 students to the group which received the experimental treatment preceding the final. The groups were well matched on sex, class standing, and ability. Scholastic Aptitude Test (SAT) scores were available for 173 of the 183 students. Mean SAT verbal scores of the two groups were not significantly different (F < I), nor were mean SAT mathematics scores significantly different (F < 1).
Materials Adjunct quesrions. Adjunct questions keyed to the introductory biology textbook (Curtis & Barnes, 1981) were written by two graduate students and edited by one of the authors (H.K.) who was also the course instructor. Approximately 2 questions per page of text were
82
SPRING,
SASSENRATH,
AND
KETELLAPPER
composed, resulting in about 800 questions. The typical adjunct question contained one or two sentences and required a constructed response of one or two words. The adjunct questions were keyed to specific pages and heading of the text, and were printed and bound in workbooks with tear-out pages. Adjunct questions covered concepts, facts, principles, and applications and dealt with text material considered most important. Approximately 30% of the adjunct questions measured verbatim recall, approximately 45% were paraphrased from the text, and approximately 25% dealt with applications not specifically covered in the text. Midterm and final examinations. Questions on the midterm and final examinations that covered textbook material were used for the criterion scores. All questions required multiple-choice answers. Following R. C. Anderson’s (1972) method of classifying questions, the questions were subdivided into verbatim-recall questions that retained the wording of material in the textbook, and comprehension questions that paraphrased material in the textbook or required students to apply that material in a manner not covered in the textbook. In addition, examination questions were subdivided along a second dimension into old and new questions. Old questions, which were intended to measure the direct effect of adjunct questions, dealt with text material that had been previously covered by adjunct questions. New questions, which were intended to measure the indirect effect of adjunct questions, dealt with text material that had not been previously covered by adjunct questions. The number of questions of each type is shown in Table 1 for the midterm and in Table 2 for the final examination.
Procedure Students assigned to the experimental treatment during the first or second halves of the course were instructed to read and answer adjunct questions immediately after reading the corresponding text material, repeating this sequence for each section of the text. It was explained to them that reading the questions before reading the corresponding material might reduce the effectiveness of the questions as a study aid. In addition, they were instructed to answer the adjunct questions from memory, and only to check the text when they were not confident of an answer. Written answers to adjunct questions were handed in weekly and were not returned to students. Compliance with instructions was determined by a questionnaire on which students indicated whether they read adjunct questions after or before reading the corresponding text material. The questionnaire was completed at the end
TABLE MEANS,
STANDARD
DEVIATIONS,
AND F VALUES MIDTERM
Midterm questions: types & numbers Old
(21)
New (25) Verbatim recall (17) Comprehension (29) Combined (46) *p
<
.05.
** p < .02.
Experimental Group 1
I
FOR PERCENT.K~E
OF CORRECT
ANSWERS
ON THE
EXAMINATION
Experimental Group 2
M
SD
M
60.9 72.5 69.6 65.7 67.2
13.8 11.9 15.0 10.8 10.8
55.2 62.9 59.5 59.3 59.4
Control grow
SD
M
SD
F(2,159)
14.9 12.4 18.4 11.8 12.7
61.5 69.0 66.1 65.3 65.6
14.1 13.9 16.4 12.6 12.5
1.96 4.91** 3.43* 2.90 3.82*
VALIDITY
OF ADJUNCT
QUESTIONS
83
of the course. The questionnaire also asked whether students in control groups studied adjunct questions obtained from students in experimental groups. Later analysis of the questionnaire indicated that no student reported this bccurring; thus it appears that the design was not violated in this way.
RESULTS
Midterm Examination Five scores were obtained from the midterm examination. There was a score for old questions and another score for new questions, regardless of whether these questions required verbatim recall or comprehension. Similarly, there was a score for verbatim-recall questions and a score for comprehension questions, regardless of whether these questions were old or new. Finally, there was a score for all questions combined. A preliminary analysis of midterm scores disclosed that students in the experimental group who indicated on the questionnaire that they followed instructions, i.e., reading and answering adjunct questions immediately after reading the relevant text material, obtained higher scores than did students who failed to follow these instructions. Accordingly, the group that followed instructions (Experimental Group 1; n = 60) was separated from the group that failed to follow instructions (Experimental Group 2; n = 23, and these two groups along with the control group (n = 77) were compared with one-way analyses of variance. Students who did not complete questionnaires were eliminated from the midterm analyses. Our first comparison indicated that the three groups were not significantly different on SAT verbal scores, F(2,159> = 1.42, p > .20; nor were they significantly different on SAT mathematics scores, F < 1. Thus, the three groups were comparable on measures of general ability. Next, although we were primarily interested in midterm questions that were taken from the textbook rather than from the lectures, we did compare the performance of the groups on lecture questions. This comparison indicated that the three groups were not significantly different on lecture-question scores, F(2,159) = 1.81, p > .20. This result provides further evidence, in addition to SAT scores, that the groups did not differ on biology-learning ability. Finally, midterm scores on textbook material were compared in the remaining analyses. These results are shown in Table 1. It may be seen that significant group differences were obtained for three of the five midterm scores: new questions, verbatim-recall questions, and combined questions. Newman-Keuls tests, with a correction for unequal sample size, were computed to further examine these differences. Contrary to our expectation, no significant difference was obtained between Experimental Group 1 and the control group. Thus, experimental students who
84
SPRING,
SASSENRATH,
AND
KETELLAPPER
followed instructions did not score significantly higher than control students. As expected, however, we did find that Experimental Group 1 scored significantly higher than Experimental Group 2. This effect was obtained for new questions, verbatim-recall questions, and combined questions (p < .05). In addition, the control group scored significantly higher than Experimental Group 2 for two of these scores: new questions and combined questions (JJ < .05). Thus, failing to follow our instruction that adjunct questions should be answered only after reading the relevant material was associated with significantly lower scores. We were particularly interested in the fact that there was a significant difference between Experimental Groups 1 and 2 for new questions but not for old questions. This pattern of results is consistent with laboratory studies which have shown that students who read adjunct questions before reading the relevant text material are primarily penalized on new questions which measure indirect learning. To determine if this pattern was significant in the present data, we compared the point-biserial correlations of new-question scores and a dummy-coded variable indicating whether experimental students followed instructions (v = .35) to the point-biserial correlation of old-question scores and the same dummycoded variable (Y = .I@. The difference between the two correlations was significant, t(82) = 1.80, p < .05, one-tailed test. Final Examination Analyses of the same five scores obtained from the final examination yielded some different but more encouraging results. As before, we first divided the experimental groups into subgroups of students who indicated on the questionnaire that they did or did not follow instructions with regard to reading and answering adjunct questions. Unlike results obtained for the midterm examination, however, we found no significant difference or trend separating these two subgroups on the final examination. The subgroups were recombined, therefore, into a single experimental group. Next, although we were primarily interested in final-examination questions that were taken from the textbook rather than from lectures, we compared scores of the experimental and control groups on lecture questions. The purpose of this comparison was to again demonstrate that, in addition to not differing significantly on SAT scores, the groups did not differ on biology-learning ability. As expected, the difference was not significant, F < 1. Finally, subsequent analyses of variance compared the performances of the experimental and control groups on each of the five scores for textbook questions on the final examination. The results are shown in
VALIDITY
OF ADJUNCT
85
QUESTIONS
Table 2. It may be seen that the mean of the experiment group was significantly higher than that of the control group for each of the five scores. It may also be seen that the effect due to adjunct questions tended to be larger for old than for new questions. For old questions, the difference between the mean scores of the experimental and control groups was 45% of the standard deviation of the control group. For new questions. this difference was only 30% of the standard deviation of the control group. Thus, consistent with most laboratory studies, the direct effect of adjunct questions tended to be larger than the indirect effect. To test the significance of this difference between direct and indirect effects, we compared the point-biserial correlation of old-question scores and a dummy-coded treatment variable (v = .21) to the point-biserial correlation of new-question scores and the same dummy-coded treatment variable (Y = .lS). The difference between the two correlations, however, was not significant, t(l80) = 1.03, p > .lO, one-tailed test. It may also be seen in Table 2 that the effect due to adjunct questions was as large for questions measuring comprehension as for those measuring verbatim recall. For comprehension questions, the difference between the mean scores of the experimental and control groups was 39% of the standard deviation of the control group. Similarly, for verbatim-recall questions this difference was 32% of the standard deviation of the control group. Thus, it appears that adjunct questions had about the same effect on comprehension and verbatim-recall questions. The failure to obtain an effect for adjunct questions on the midterm examination suggest an alternative interpretation of the positive effect that was obtained on the final examination. It is possible that the apparent positive effect was due to a drop in the performance of students in the second-half control group, rather than to an improvement in the perTABLE
2
MEANS, STANDARD DEVIATIONS, ,\ND F VALUES PERCENTAGE OF CORRECT ANSWERS ON THE FINAL
Questions: types & numbers Old (25) New (24) Verbatim recall (17) Comprehension (32) Combined (49) * p < .05.
** p < .02.
Experimental
M 66.7 63.7 71.6 61.8 65.2
FOR
EXAMINATION
Group
Control group
SD
M
SD
15.0 15.7 13.7 16.3 13.8
60.5 59.1 67.2 56.0 59.9
13.8 15.2 13.8 14.9 13.1
F(l,lXl) 8.25** 3.92* 4.71* 6.33** 7.16**
86
SPRING,
SASSENRATH,
AND
KETELLAPPER
formance of students in the second-half experimental group. Such a drop might result from an unintended modification of textbook study habits when group assignments were reversed midway through the course. To rule out this alternative interpretation, we compared the second-half experimental and control groups to all students who took the course during the previous year. The comparison was limited to 18 questions which were common to the final examinations given, but not returned to the students, in both years. It was found that, compared to the previous class, the control group performed better on 9 questions and worse on the remaining 9 questions. Thus, there was no evidence that the control group performed differently, on the final examination, than the previous class. The experimental group, on the other hand, obtained higher mean scores than the previous class on 14 of the 18 questions. The probability of this result occurring by chance is only p = .015, one-tailed test. These results strengthen our confidence that the advantage of students in the experimental group, rather than being an artifact of the crossover experimental design, was due to the hypothesized adjunct-questions effect. DISCUSSION
The present results lead us to a position of guarded optimism concerning the value of adjunct questions in reading a science textbook densely packed with material which students must retain and understand. Results obtained in the second half of the course were as expected. On all measures of textbook learning, the experimental group earned significantly higher scores than the control group. Equally important, the differences were of practical as well as statistical significance. For example, for all types of textbook questions combined, the difference favoring the experimental group was 40% of the standard deviation of the control group. In our experience, it is not easy to achieve an effect of this moderate magnitude in real-life instructional settings. Finally, a significant effect was obtained not only in a comparison of the experimental and control groups, but also in a comparison of the experimental group with students who took the course the previous year. It is also noteworthy that this effect was detected with a final examination composed exclusively of multiple-choice questions. Based on previous laboratory studies, there is reason to believe that a larger treatment effect might have been obtained if the examination had contained a significant proportion of questions requiring constructed answers (R. C. Anderson & Myrow, 1971; Roderick & Anderson, 1968). Furthermore, the positive effect obtained on the final examination was as large for comprehension questions as for verbatim-recall questions. Since our goal is to improve meaningful learning rather than to merely
VALIDITY
OF ADJUNCT
QUESTIONS
87
improve the verbatim recall of textbook material, this result is particularly encouraging. Also of interest was the fact that a significant treatment effect was obtained on the final examination for new questions (indirect effects) as well as for old questions (direct effects). As previously noted, laboratory studies of adjunct questions have usually obtained considerably larger direct effects than indirect effects. In a review of these studies, Ladas (1973) concluded that the magnitude of the facilitative effect of adjunct questions on the indirect learning of information not covered by the questions, if it exists at all, is probably not large. In the present study, however, the effect for old questions (direct effect) was not significantly larger than the effect for new questions (indirect effect), and indirect as well as direct learning was improved. It is possible that this desirable result is due to the use of biology material in which concepts, facts, and principles are related to an unusual degree, and in which an improvement of memory for material directly covered by adjunct questions might be expected also to improve memory for related material which was not covered directly. In addition, it is possible that obtaining a significant indirect effect was also due, in part, to the high density of adjunct questions that was used. If a lower density had been used, say one question rather than two questions per textbook page, the temporal contiguity of material that was covered and not covered by adjunct questions would have been reduced. Similarly, the topical relatedness of material that was covered and not covered by adjunct questions would have been reduced. Under these conditions it is possible that the direct effect would also have been reduced. This would be expected if answering an adjunct question not only increases the strength of a memory-retrieval pathway to information contained in the question (Gagne, 1978), but also increases the number of pathways leading to information that is closely related, temporally or topically, to the question. According to J. R. Anderson (1976), increasing the number of pathways leading to a concept should increase its retrievability. We also obtained less encouraging but important results on the midterm examination. First, lower midterm scores were earned by students who ignored our instructions and read adjunct questions before reading the relevant text material, although these same students were not significantly deficient on measures of general ability or on a measure of biology-learning ability. Not only did they score lower than experimental students who followed our instructions, but they also scored lower than students in the control group. This deficit was significant on new questions which measured indirect learning, but not on old questions which measured direct learning. As previously mentioned, a similar pattern has been found in laboratory studies in which question placement, before or
88
SPRING,
SASSENRATH,
AND
KETELLAPPER
after the relevant text, has been manipulated (R. C. Anderson & Biddle, 1975). In the present study, students who did not follow instructions could have used the adjunct questions as an opportunity to look up answers rather than carefully read an entire assignment. This superficial approach to studying, of course, would affect performance on new examination questions more than on old examination questions. Also of concern on the midterm examination was the failure to find, even for students in the experimental group who followed our instructions, a facilitative effect of adjunct questions. We suspect that this inconsistency between the midterm and final examinations may be due to a change in the way students studied following the midterm examination. The low scores on the midterm could have led students to the sobering conclusion that the course was more difficult than they had anticipated. This realization, which might have been especially strong in an introductory course for nonmajors, could have changed the way students used adjunct questions during the second half of the course. Thus, in the first half, students who followed instructions, reading adjunct questions after reading the text, may have failed to read as carefully as they might have if the questions had not been available, relying instead on their later reading of adjunct questions to identify the most important concepts, facts, and principles. In the second half, on the other hand, corresponding students prior to answering adjunct questions, may have, read the text as carefully as students in the control group. There is no way of knowing if this interpretation is correct. If it is, our results underscore the possibility that adjunct questions, if they induce some students to read less carefully than they would without them, may actually hurt rather than help learning. It is possible that if we had spent more time justifying and explaining how to study with adjunct questions, and had given a stronger warning about their possible misuse, positive results might have been obtained in the first as well as the second half of the course. This interpretation is consistent with the view that students should be informed participants in any program to introduce a new learning method (Brown, Campione, & Day, 1981). The present results suggest that rather than requiring all students to use adjunct questions, it would be better to make them optional. Presumably, adjunct questions would be used, under an optional condition, only by students who were motivated to use them properly after receiving strong warnings concerning their possible misuse. REFERENCES J. R. (1976). Lungungu, memory and rhought. Hillsdale, NJ: Erlbaum. R. C. (1972). How to construct achievement tests to assess comprehension. viens of Education Research, 42, 145-170.
ANDERSON, ANDERSON,
RP-
VALIDITY
OF ADJUNCT
QUESTIONS
89
R. C., & BIDDLE, W. B. (1975). On asking people questions about what they are reading. In G. H. Bower (Ed.), The psychology oj’ learning and morivarion (Vol. 9). New York: Academic Press. ANDERSONR. C., & MYROW, D. L. (1971). Retroactive inhibition of meaningful discourse. ANDERSON,
Journal
of Educational
Psychology
Monograph,
62, 81-94.
A. L., CAMPIONE, J. C., & DAY, I. D. (1981). Learning to learn: On training students to learn from texts. Educational Researcher. 10, 14-21. CURTIS, H., & BARNES,N. S. (1981). Invitation 10 biology (3rd ed.). New York: Worth Publishers. FAW, H. W., & WALLER, T. G. (1976). Mathemagenic behaviors and efficiency in learning from prose material: Review, critique and recommendations. Review of Educational BROWN,
Research,
46, 69 I-720.
GACNE, E. D. (1978). Long-term Review
of Educational
retention of information following learning from prose.
Research,
48, 629-W.
LADAS, H. (1973). The mathemagenic effect of factual review questions on the learning of incidental information: A critical review. Revietij of Educational Research, 43, 71-82. RODERICK, M. C., & ANDERSON, R. C. (1968). A programmed introduction to psychology versus a textbook-style summary of the same lesson. Journal of Educational Psychology,
59, 381-387.
E. Z. (1965). Some theoretical and experimental approaches to problems in written instruction. In J. D. Krumboltz (Ed.), Learning and the educational process. Chicago: Rand McNally.
ROTHKOPF,
EDUCATIONAL
Ecological
PSYCHOLOGY
Validity
CARL SPRING, JULIUS
11,
79-89 (1986)
of Adjunct Questions Biology Course
in a College
SASSENRATH,
KETELLAPPER
lini~~rsity
AND HENDRIK
of Ca/iforni~~,
Dn\~is
To validate their use in a natural environment with realistic learning materials and conditions, the efficacy of adjunct questions was tested in a college biology course with a double-crossover design. Mixed results were obtained. On the one hand, students using adjunct questions during the second half of the course scored significantly higher than control students on a variety of measures obtained on the final examination. Significant indirect as well as direct effects were obtained, and significant effects were demonstrated on comprehension as well as verbatim-recall examination questions. On the other hand, during the first half of the course evidence was also obtained that some students did not follow instructions on the proper use of adjunct questions, and that for those students adjunct questions actually impaired learning. Such vagaries may be common in natural experimental settings as opposed to highly controlled laboratory settings. It is concluded that, rather than requiring all students to use adjunct questions. it would be better to make them optional. Presumably, adjunct questions would be used. under an optional condition, only by students who were motivated to use them properly after receiving strong warnings concerning their possible misuse. ‘C IOXh Academc Press. 1°C
A good deal of research has been directed toward the study of instructional procedures which, although promising, never find their way into classrooms. Research on the use of adjunct questions to improve learning is a case in point. Adjunct questions are questions which are inserted at critical points within a text to facilitate a student’s understanding and retention of the material. Starting with the seminal work of Rothkopf (1965), there have been a large number of experimental studies that have investigated the use of adjunct questions. The frequency of these investigations peaked during the last decade and is now in decline. Although most of these studies demonstrated that adjunct questions can facilitate learning (R. C. Anderson & Biddle, 1973, this research has had little influence on educational practice. One possible reason for this gap between research and practice is that instructional studies are often conducted under unnaturally controlled Financial assistance for this research came from the Basic Skills Research Program, University of California, Davis. We thank Bruce Campell and Daryl Koutnik for writing the adjunct questions, and Sam Neustadt for assistance in analyzing the data. Send requests for
reprints to Dr. Julius Sassenrath, Department of Education, University of California. Davis, CA 95616. 19 0361-476X186 $3.00 Copyright ‘c 1986 by Academic Preaa. Inc. All rights of reproduction in any form revxve4
80
SPRING,
SASSENRATH,
AND
KETELLAPPER
conditions and for unrealistically short durations. They are perceived by practitioners, therefore, as having insufficient ecological validity. This is especially true of research on adjunct questions. Although the effects of adjunct questions are well understood under laboratory controlled conditions with short, specially constructed reading passages, their effects with relatively uncontrolled homework reading of a large textbook are not well understood (Faw & Waller, 1976). To correct this problem, the present study was conducted in a college biology course under conditions which differed from typical laboratory studies in several ways that should improve its ecological validity: (I) the duration of the study was much longer, (2) the reading assignments were considerably larger, (3) the material to be learned was more realistic and more conceptually integrated, and (4) it was not possible to monitor the students as they read the textbook and answered adjunct questions. Because our objective was primarily applied rather than theoretical, we used several findings from laboratory studies in an attempt to maximize the probable impact of the experimental treatment. For example, it is known that adjunct questions presented after reading relevant sections of a text are more effective than questions presented before the relevant text, and that this difference is larger for text material not specifically covered by adjunct questions (indirect effect) than for material covered by adjunct questions (direct effect) (R. C. Anderson & Biddle, 1975). Also these authors point out that some studies have shown that reading adjunct questions before reading the text may actually inhibit indirect learning. In the present study, therefore, students in the experimental groups were instructed to read and answer adjunct questions after rather than before reading relevant text material. Also from the review by R. C. Anderson and Biddle (1975) we know that (1) adjunct questions requiring overt written responses are more effective than those requiring covert mental responses, (2) adjunct questions requiring constructed answers are more effective than those requiring multiple-choice answers, and (3) answering adjunct questions after reading one or two paragraphs is more effective than answering them after reading longer sections of text. These results from laboratory studies were also used in designing the experimental treatment so as to maximize its probable impact. In addition to our general objective of maximizing the impact of the experimental treatment, we had several other objectives. First, we wanted to compare the direct and indirect effects of answering adjunct questions. In most laboratory studies the direct effects are considerably larger than the indirect effects, and in some studies the indirect effects have been negligible (R. C. Anderson & Biddle, 1975). We reasoned, however, that with the highly integrated material covered in a biology textbook, answering an adjunct question might improve one’s memory
VALIDITY
OF ADJUNCT
QUESTIONS
81
for concepts that are related to the question as well as for concepts that are actually covered by the question. In this case, we would expect a significant indirect effect as well as a direct effect. Second, we wanted to determine whether students would, as instructed, read adjunct questions after rather than before reading the relevant text material. This was determined with a questionnaire administered at the end of the course. Related to this, we also wanted to measure the direct and indirect effects of failing to follow instructions. Based on laboratory evidence discussed above, it was expected that not following instructions would reduce the indirect effect more than the direct effect. Finally, we wanted to determine whether adjunct questions are as effective in improving comprehension as they are in improving verbatim recall. To determine this, we included both types of questions on the midterm and final examinations from which the effects of the experimental treatment were measured. METHOD
Design and Subjects The one-quarter introductory biology course was for students at the University of California, Davis Campus, not majoring in a biological science. The course was divided into two sessions of equal duration. The first session was followed by a midterm multiple-choice examination on material covered during the first half of the course. The second session was followed by a final multiple-choice examination, a large portion of which covered material from the second half of the course. During the first half of the course, about half of the students were required to complete adjunct questions (experimental group), and the other half were not (control group). During the second half of the course, group assignments were reversed, resulting in an AB, BA design. The primary reason for using the crossover design was to be fair to students in the class by having them experience both the treatment and control conditions. The mean scores of the experimental and control groups were compared on the midterm examination, but only on questions covering material from the textbook. After group assignments were reversed halfway through the course, the mean scores of the new experimental and control groups were again compared on the final examination. For the latter comparison, however, only questions on the final examination which covered material presented during the second half of the course from the textbook were considered. For practical reasons, all students within a discussion section were assigned to the same group, and discussion sections of approximately 20 students each were randomly assigned to the two groups. This resulted in the assignment of 97 students to the group which received the experimental treatment preceding the midterm and 86 students to the group which received the experimental treatment preceding the final. The groups were well matched on sex, class standing, and ability. Scholastic Aptitude Test (SAT) scores were available for 173 of the 183 students. Mean SAT verbal scores of the two groups were not significantly different (F < I), nor were mean SAT mathematics scores significantly different (F < 1).
Materials Adjunct quesrions. Adjunct questions keyed to the introductory biology textbook (Curtis & Barnes, 1981) were written by two graduate students and edited by one of the authors (H.K.) who was also the course instructor. Approximately 2 questions per page of text were
82
SPRING,
SASSENRATH,
AND
KETELLAPPER
composed, resulting in about 800 questions. The typical adjunct question contained one or two sentences and required a constructed response of one or two words. The adjunct questions were keyed to specific pages and heading of the text, and were printed and bound in workbooks with tear-out pages. Adjunct questions covered concepts, facts, principles, and applications and dealt with text material considered most important. Approximately 30% of the adjunct questions measured verbatim recall, approximately 45% were paraphrased from the text, and approximately 25% dealt with applications not specifically covered in the text. Midterm and final examinations. Questions on the midterm and final examinations that covered textbook material were used for the criterion scores. All questions required multiple-choice answers. Following R. C. Anderson’s (1972) method of classifying questions, the questions were subdivided into verbatim-recall questions that retained the wording of material in the textbook, and comprehension questions that paraphrased material in the textbook or required students to apply that material in a manner not covered in the textbook. In addition, examination questions were subdivided along a second dimension into old and new questions. Old questions, which were intended to measure the direct effect of adjunct questions, dealt with text material that had been previously covered by adjunct questions. New questions, which were intended to measure the indirect effect of adjunct questions, dealt with text material that had not been previously covered by adjunct questions. The number of questions of each type is shown in Table 1 for the midterm and in Table 2 for the final examination.
Procedure Students assigned to the experimental treatment during the first or second halves of the course were instructed to read and answer adjunct questions immediately after reading the corresponding text material, repeating this sequence for each section of the text. It was explained to them that reading the questions before reading the corresponding material might reduce the effectiveness of the questions as a study aid. In addition, they were instructed to answer the adjunct questions from memory, and only to check the text when they were not confident of an answer. Written answers to adjunct questions were handed in weekly and were not returned to students. Compliance with instructions was determined by a questionnaire on which students indicated whether they read adjunct questions after or before reading the corresponding text material. The questionnaire was completed at the end
TABLE MEANS,
STANDARD
DEVIATIONS,
AND F VALUES MIDTERM
Midterm questions: types & numbers Old
(21)
New (25) Verbatim recall (17) Comprehension (29) Combined (46) *p
<
.05.
** p < .02.
Experimental Group 1
I
FOR PERCENT.K~E
OF CORRECT
ANSWERS
ON THE
EXAMINATION
Experimental Group 2
M
SD
M
60.9 72.5 69.6 65.7 67.2
13.8 11.9 15.0 10.8 10.8
55.2 62.9 59.5 59.3 59.4
Control grow
SD
M
SD
F(2,159)
14.9 12.4 18.4 11.8 12.7
61.5 69.0 66.1 65.3 65.6
14.1 13.9 16.4 12.6 12.5
1.96 4.91** 3.43* 2.90 3.82*
VALIDITY
OF ADJUNCT
QUESTIONS
83
of the course. The questionnaire also asked whether students in control groups studied adjunct questions obtained from students in experimental groups. Later analysis of the questionnaire indicated that no student reported this bccurring; thus it appears that the design was not violated in this way.
RESULTS
Midterm Examination Five scores were obtained from the midterm examination. There was a score for old questions and another score for new questions, regardless of whether these questions required verbatim recall or comprehension. Similarly, there was a score for verbatim-recall questions and a score for comprehension questions, regardless of whether these questions were old or new. Finally, there was a score for all questions combined. A preliminary analysis of midterm scores disclosed that students in the experimental group who indicated on the questionnaire that they followed instructions, i.e., reading and answering adjunct questions immediately after reading the relevant text material, obtained higher scores than did students who failed to follow these instructions. Accordingly, the group that followed instructions (Experimental Group 1; n = 60) was separated from the group that failed to follow instructions (Experimental Group 2; n = 23, and these two groups along with the control group (n = 77) were compared with one-way analyses of variance. Students who did not complete questionnaires were eliminated from the midterm analyses. Our first comparison indicated that the three groups were not significantly different on SAT verbal scores, F(2,159> = 1.42, p > .20; nor were they significantly different on SAT mathematics scores, F < 1. Thus, the three groups were comparable on measures of general ability. Next, although we were primarily interested in midterm questions that were taken from the textbook rather than from the lectures, we did compare the performance of the groups on lecture questions. This comparison indicated that the three groups were not significantly different on lecture-question scores, F(2,159) = 1.81, p > .20. This result provides further evidence, in addition to SAT scores, that the groups did not differ on biology-learning ability. Finally, midterm scores on textbook material were compared in the remaining analyses. These results are shown in Table 1. It may be seen that significant group differences were obtained for three of the five midterm scores: new questions, verbatim-recall questions, and combined questions. Newman-Keuls tests, with a correction for unequal sample size, were computed to further examine these differences. Contrary to our expectation, no significant difference was obtained between Experimental Group 1 and the control group. Thus, experimental students who
84
SPRING,
SASSENRATH,
AND
KETELLAPPER
followed instructions did not score significantly higher than control students. As expected, however, we did find that Experimental Group 1 scored significantly higher than Experimental Group 2. This effect was obtained for new questions, verbatim-recall questions, and combined questions (p < .05). In addition, the control group scored significantly higher than Experimental Group 2 for two of these scores: new questions and combined questions (JJ < .05). Thus, failing to follow our instruction that adjunct questions should be answered only after reading the relevant material was associated with significantly lower scores. We were particularly interested in the fact that there was a significant difference between Experimental Groups 1 and 2 for new questions but not for old questions. This pattern of results is consistent with laboratory studies which have shown that students who read adjunct questions before reading the relevant text material are primarily penalized on new questions which measure indirect learning. To determine if this pattern was significant in the present data, we compared the point-biserial correlations of new-question scores and a dummy-coded variable indicating whether experimental students followed instructions (v = .35) to the point-biserial correlation of old-question scores and the same dummycoded variable (Y = .I@. The difference between the two correlations was significant, t(82) = 1.80, p < .05, one-tailed test. Final Examination Analyses of the same five scores obtained from the final examination yielded some different but more encouraging results. As before, we first divided the experimental groups into subgroups of students who indicated on the questionnaire that they did or did not follow instructions with regard to reading and answering adjunct questions. Unlike results obtained for the midterm examination, however, we found no significant difference or trend separating these two subgroups on the final examination. The subgroups were recombined, therefore, into a single experimental group. Next, although we were primarily interested in final-examination questions that were taken from the textbook rather than from lectures, we compared scores of the experimental and control groups on lecture questions. The purpose of this comparison was to again demonstrate that, in addition to not differing significantly on SAT scores, the groups did not differ on biology-learning ability. As expected, the difference was not significant, F < 1. Finally, subsequent analyses of variance compared the performances of the experimental and control groups on each of the five scores for textbook questions on the final examination. The results are shown in
VALIDITY
OF ADJUNCT
85
QUESTIONS
Table 2. It may be seen that the mean of the experiment group was significantly higher than that of the control group for each of the five scores. It may also be seen that the effect due to adjunct questions tended to be larger for old than for new questions. For old questions, the difference between the mean scores of the experimental and control groups was 45% of the standard deviation of the control group. For new questions. this difference was only 30% of the standard deviation of the control group. Thus, consistent with most laboratory studies, the direct effect of adjunct questions tended to be larger than the indirect effect. To test the significance of this difference between direct and indirect effects, we compared the point-biserial correlation of old-question scores and a dummy-coded treatment variable (v = .21) to the point-biserial correlation of new-question scores and the same dummy-coded treatment variable (Y = .lS). The difference between the two correlations, however, was not significant, t(l80) = 1.03, p > .lO, one-tailed test. It may also be seen in Table 2 that the effect due to adjunct questions was as large for questions measuring comprehension as for those measuring verbatim recall. For comprehension questions, the difference between the mean scores of the experimental and control groups was 39% of the standard deviation of the control group. Similarly, for verbatim-recall questions this difference was 32% of the standard deviation of the control group. Thus, it appears that adjunct questions had about the same effect on comprehension and verbatim-recall questions. The failure to obtain an effect for adjunct questions on the midterm examination suggest an alternative interpretation of the positive effect that was obtained on the final examination. It is possible that the apparent positive effect was due to a drop in the performance of students in the second-half control group, rather than to an improvement in the perTABLE
2
MEANS, STANDARD DEVIATIONS, ,\ND F VALUES PERCENTAGE OF CORRECT ANSWERS ON THE FINAL
Questions: types & numbers Old (25) New (24) Verbatim recall (17) Comprehension (32) Combined (49) * p < .05.
** p < .02.
Experimental
M 66.7 63.7 71.6 61.8 65.2
FOR
EXAMINATION
Group
Control group
SD
M
SD
15.0 15.7 13.7 16.3 13.8
60.5 59.1 67.2 56.0 59.9
13.8 15.2 13.8 14.9 13.1
F(l,lXl) 8.25** 3.92* 4.71* 6.33** 7.16**
86
SPRING,
SASSENRATH,
AND
KETELLAPPER
formance of students in the second-half experimental group. Such a drop might result from an unintended modification of textbook study habits when group assignments were reversed midway through the course. To rule out this alternative interpretation, we compared the second-half experimental and control groups to all students who took the course during the previous year. The comparison was limited to 18 questions which were common to the final examinations given, but not returned to the students, in both years. It was found that, compared to the previous class, the control group performed better on 9 questions and worse on the remaining 9 questions. Thus, there was no evidence that the control group performed differently, on the final examination, than the previous class. The experimental group, on the other hand, obtained higher mean scores than the previous class on 14 of the 18 questions. The probability of this result occurring by chance is only p = .015, one-tailed test. These results strengthen our confidence that the advantage of students in the experimental group, rather than being an artifact of the crossover experimental design, was due to the hypothesized adjunct-questions effect. DISCUSSION
The present results lead us to a position of guarded optimism concerning the value of adjunct questions in reading a science textbook densely packed with material which students must retain and understand. Results obtained in the second half of the course were as expected. On all measures of textbook learning, the experimental group earned significantly higher scores than the control group. Equally important, the differences were of practical as well as statistical significance. For example, for all types of textbook questions combined, the difference favoring the experimental group was 40% of the standard deviation of the control group. In our experience, it is not easy to achieve an effect of this moderate magnitude in real-life instructional settings. Finally, a significant effect was obtained not only in a comparison of the experimental and control groups, but also in a comparison of the experimental group with students who took the course the previous year. It is also noteworthy that this effect was detected with a final examination composed exclusively of multiple-choice questions. Based on previous laboratory studies, there is reason to believe that a larger treatment effect might have been obtained if the examination had contained a significant proportion of questions requiring constructed answers (R. C. Anderson & Myrow, 1971; Roderick & Anderson, 1968). Furthermore, the positive effect obtained on the final examination was as large for comprehension questions as for verbatim-recall questions. Since our goal is to improve meaningful learning rather than to merely
VALIDITY
OF ADJUNCT
QUESTIONS
87
improve the verbatim recall of textbook material, this result is particularly encouraging. Also of interest was the fact that a significant treatment effect was obtained on the final examination for new questions (indirect effects) as well as for old questions (direct effects). As previously noted, laboratory studies of adjunct questions have usually obtained considerably larger direct effects than indirect effects. In a review of these studies, Ladas (1973) concluded that the magnitude of the facilitative effect of adjunct questions on the indirect learning of information not covered by the questions, if it exists at all, is probably not large. In the present study, however, the effect for old questions (direct effect) was not significantly larger than the effect for new questions (indirect effect), and indirect as well as direct learning was improved. It is possible that this desirable result is due to the use of biology material in which concepts, facts, and principles are related to an unusual degree, and in which an improvement of memory for material directly covered by adjunct questions might be expected also to improve memory for related material which was not covered directly. In addition, it is possible that obtaining a significant indirect effect was also due, in part, to the high density of adjunct questions that was used. If a lower density had been used, say one question rather than two questions per textbook page, the temporal contiguity of material that was covered and not covered by adjunct questions would have been reduced. Similarly, the topical relatedness of material that was covered and not covered by adjunct questions would have been reduced. Under these conditions it is possible that the direct effect would also have been reduced. This would be expected if answering an adjunct question not only increases the strength of a memory-retrieval pathway to information contained in the question (Gagne, 1978), but also increases the number of pathways leading to information that is closely related, temporally or topically, to the question. According to J. R. Anderson (1976), increasing the number of pathways leading to a concept should increase its retrievability. We also obtained less encouraging but important results on the midterm examination. First, lower midterm scores were earned by students who ignored our instructions and read adjunct questions before reading the relevant text material, although these same students were not significantly deficient on measures of general ability or on a measure of biology-learning ability. Not only did they score lower than experimental students who followed our instructions, but they also scored lower than students in the control group. This deficit was significant on new questions which measured indirect learning, but not on old questions which measured direct learning. As previously mentioned, a similar pattern has been found in laboratory studies in which question placement, before or
88
SPRING,
SASSENRATH,
AND
KETELLAPPER
after the relevant text, has been manipulated (R. C. Anderson & Biddle, 1975). In the present study, students who did not follow instructions could have used the adjunct questions as an opportunity to look up answers rather than carefully read an entire assignment. This superficial approach to studying, of course, would affect performance on new examination questions more than on old examination questions. Also of concern on the midterm examination was the failure to find, even for students in the experimental group who followed our instructions, a facilitative effect of adjunct questions. We suspect that this inconsistency between the midterm and final examinations may be due to a change in the way students studied following the midterm examination. The low scores on the midterm could have led students to the sobering conclusion that the course was more difficult than they had anticipated. This realization, which might have been especially strong in an introductory course for nonmajors, could have changed the way students used adjunct questions during the second half of the course. Thus, in the first half, students who followed instructions, reading adjunct questions after reading the text, may have failed to read as carefully as they might have if the questions had not been available, relying instead on their later reading of adjunct questions to identify the most important concepts, facts, and principles. In the second half, on the other hand, corresponding students prior to answering adjunct questions, may have, read the text as carefully as students in the control group. There is no way of knowing if this interpretation is correct. If it is, our results underscore the possibility that adjunct questions, if they induce some students to read less carefully than they would without them, may actually hurt rather than help learning. It is possible that if we had spent more time justifying and explaining how to study with adjunct questions, and had given a stronger warning about their possible misuse, positive results might have been obtained in the first as well as the second half of the course. This interpretation is consistent with the view that students should be informed participants in any program to introduce a new learning method (Brown, Campione, & Day, 1981). The present results suggest that rather than requiring all students to use adjunct questions, it would be better to make them optional. Presumably, adjunct questions would be used, under an optional condition, only by students who were motivated to use them properly after receiving strong warnings concerning their possible misuse. REFERENCES J. R. (1976). Lungungu, memory and rhought. Hillsdale, NJ: Erlbaum. R. C. (1972). How to construct achievement tests to assess comprehension. viens of Education Research, 42, 145-170.
ANDERSON, ANDERSON,
RP-
VALIDITY
OF ADJUNCT
QUESTIONS
89
R. C., & BIDDLE, W. B. (1975). On asking people questions about what they are reading. In G. H. Bower (Ed.), The psychology oj’ learning and morivarion (Vol. 9). New York: Academic Press. ANDERSONR. C., & MYROW, D. L. (1971). Retroactive inhibition of meaningful discourse. ANDERSON,
Journal
of Educational
Psychology
Monograph,
62, 81-94.
A. L., CAMPIONE, J. C., & DAY, I. D. (1981). Learning to learn: On training students to learn from texts. Educational Researcher. 10, 14-21. CURTIS, H., & BARNES,N. S. (1981). Invitation 10 biology (3rd ed.). New York: Worth Publishers. FAW, H. W., & WALLER, T. G. (1976). Mathemagenic behaviors and efficiency in learning from prose material: Review, critique and recommendations. Review of Educational BROWN,
Research,
46, 69 I-720.
GACNE, E. D. (1978). Long-term Review
of Educational
retention of information following learning from prose.
Research,
48, 629-W.
LADAS, H. (1973). The mathemagenic effect of factual review questions on the learning of incidental information: A critical review. Revietij of Educational Research, 43, 71-82. RODERICK, M. C., & ANDERSON, R. C. (1968). A programmed introduction to psychology versus a textbook-style summary of the same lesson. Journal of Educational Psychology,
59, 381-387.
E. Z. (1965). Some theoretical and experimental approaches to problems in written instruction. In J. D. Krumboltz (Ed.), Learning and the educational process. Chicago: Rand McNally.
ROTHKOPF,