- Email: [email protected]
The processing of informative feedback about multiple-choice test performance
CONTEMPORARY
EDUCATIONAL
4, 381-394 (1979)
PSYCHOLOGY
The Processing of Informative Feedback Multiple-Choice Test Performance
about
GARY D. PHYE Iotzw
Stare
Univcmity
From a cognitive perspective, informative feedback is assumed to be confirmatory or corrective. However, feedback is not always effective and errors persist. In cases where errors persist, an analysis of errors as well as correct responses is recommended in order to better understand the processing of informative feedback. A model of multiple-choice processing is introduced to provide the context for a consideration of the role of error analysis. Data from experiment 1 demonstrate the use of semantic retrieval cues in multiple-choice performance. While informative feedback improves post-test performance, it is difficult to demonstrate a differential effect for type of feedback. Experiment 2 suggests a threshold effect for informative feedback. Information beyond that which is necessary for the confirmation of a correct response and/or the correction of an error has a deleterious effect on post-test performance. Error analysis indicated a disproportionate number of perseverative errors when the use of a retrieval process was not required at post-test.
Since the early work of E. L. Thorndike, it has been assumed that informative feedback in the form of knowledge of results facilitates performance. Whether the improvement in performance is the result of motivational or informational consequences depends in part on the type of task under consideration. When the task is verbal and meets the definition of meaningful learning frequently seen in the classroom, the informational consequences appear to predominate. For example, Kulhavy and Anderson (1972), Phye, Gugliemella, and Sola (1976), and Sassenrath (1975) have marshaled evidence for the role of informative feedback in the facilitation of classroom learning through the correction of previous errors. In the classroom, a multiple-choice (M-C) format is frequently used to test abilities, skills, and knowledge as attested to by its extensive use on teacher-made tests as well as standardized achievement and aptitude tests. In the experimental literature, the M-C format has been used to measure the comprehension and retention of classroom materials following informative feedback (Phye & Baller, 1970; Sassenrath & Garverick, 1965; Suber & Anderson, 1975). Also, Anderson (1972) has recommended the use of a multiple-choice format with paraphrasing in order to measure Based on the paper “The role of informative feedback in productive learning” presented at the American Educational Research Association annual meeting, New York, 1977. Appreciation is extended to Joseph Gugliemella for his help with the data collection in experiment 1. 381 0361-476X/79/040381-14$02.00/0 Copyright Q 1979 by Academic Press, Inc. All rights of reproduction in any form reserved.
382
GARY
D. PHYE
the comprehension of meaningful material. Meaningful learning is here considered to be semantic in nature. Specifically, semantic memory (Tulving, 1972) is the organized knowledge a person possesses about words, meanings, relations, concepts, rules, etc. Input into semantic memory is processed and related to existing cognitive structures. In addition to defining the type of learning to be studied and the type of retention measure used, a conceptualization of the nature and role of informative feedback is necessary. In keeping with the semantic nature of the present definition of meaningful learning, the nature of informative feedback is herein considered in terms of its semantic content. Consequently, feedback could take the form of words, rules, concepts, meanings, etc. Hence, the role of feedback is either to provide confirmation of existing knowledge structures and schemes or to provide the semantic basis for the modification of an existing structure or scheme. Having specified a M-C format as the retention measure, consideration will be directed toward the development of a theoretical model of multiple-choice performance from a cognitive perspective. The M-C model is presented in order to provide a context for a consideration of errors following informative feedback. The error analysis that will be proposed provides the basis for identifying breakdowns in the processing of informative feedback. In light of the recent emphasis on the investigation of cognitive processes involved in instructional situations (Klahr, 1976), and an increased utilization of information processing theory, an examination of theoretical assumptions is essential when examining the role of informative feedback in instruction. Cognitive Perspective “A cognitive approach implies that learning from instruction is scientifically more productively studied as an internally, cognitively mediated process than as a direct product of the environment, people, or factors external to the learner” (Wittrock, 1978, p. 15). This view is consistent with that expressed by Carroll (1976). Cognitive theory would assert that “information is what is learned and would object to behavioral theory’s postulation that it is the responses that are learned.” “According to cognitive theory, learning to make particular responses is an internal process, as is also the decision to emit them on a particular occasion and under particular conditions.” A corollary assumption is that “feedback and correction, in cognitive theory, would be regarded as merely one kind of information contributing to learning-though frequently an important kind of information” (Carroll, 1976, p. 17). Also, there have been recent modifications in the theoretical conceptualization of the recognition process. Adams (1976) has pointed out that, while it is apparent that we search our memory at recall, it is not readily
THE PROCESSING
OF INFORMATIVE
FEEDBACK
383
apparent that search and retrieval operate for recognition. Clarification on this point has been offered by Sternberg (1966, 1969) who presents evidence for the consideration of search and retrieval as factors in recognition. Also, within the context of Human Associate Memory (HAM), Anderson and Bower (1973) consider the recognition of classes, individuals, and familiar patterns (perceptual concepts) with or without retrieval of context. These considerations of the processing involved in the recognition process would suggest a reconceptualization of multiple-choice test performance. Multiple-Choice Test Performance In order to consider the role of informative feedback when learning is assessedusing a M-C format, a M-C processing model has been proposed (Phye, Note 1). As indicated in Fig. 1, the model incorporates the cognitive assumption that instances of both storage and retrieval from memory are included in M-C performance. The model assumes several stages of processing and involves more than a simple single-stage discrimination process. A major reason for the inability to account for M-C performance without the consideration of a retrieval process is that the stem of a test item may serve to direct attention or serve as a retrieval cue for the establishment of a semantic context. For instance, when initially encountering a multiple-choice test item, a learner will typically read the stem followed by a reading of the alternatives. A choice without retrieval of context may be possible if the stem furnishes grammatical or syntactical cues that can be used in selecting an alternative. If this situation occurs, the subject goes to the next item. If not, the subject will likely use the stem as a retrieval cue in order to establish the context for memory search. If the stem has not served as a cue for the retrieval of context, the subject will guess at this point and go to the next item. In addition to the use of the stem as a retrieval cue in order to establish context, the stem may also serve as a cue for the retrieval of classes of information within a context. For example, if the stem dealt with classroom management and the promotion of consistent responding, the context would be operant conditioning principles and the class would be schedules of reinforcement. The assumption is made that the retrieval of context is a necessary but not sufficient condition for class retrieval. If the item stem can be used to retrieve both context and a class of information, the subject can focus on alternatives. Having previously narrowed memory search to a class of information, that information can be used to eliminate alternatives that fall outside the class definition. The basis for the elimination of alternatives would be the observed incongruity between the memory schema defined in terms of context and class and the alternatives contained in the
384
GARY D. PHYE
No-
-YSt FIG. 1. Processing model of multiple-choice test performance.
item. A subject may stop at this point and select an alternative. However, the subject may persevere and engage in one more stage of processing. Having narrowed the search to a class within a context, a specific item or instance from within a class may be retrieved and serve as the basis for a match between a specific memory representation and remaining alternatives. In some cases, a subject need not go through all of the stages in a serial manner. There are instances where it must be assumed that a person could bypass the class search and go directly from context to a specific memory representation. This situation would most likely exist when the item was very meaningful. The present model would suggest that the high level of correct response performance typically associated with M-C retention measures is not due to the simplicity (i.e., storage but not retrieval) of the required processing. Rather, the high level of performance appears to be due to the abundance of retrieval cues and their use in selecting an alternative. Informative
Feedback and M-C Performance
Informative feedback can serve two functions. In the case of a correct answer, feedback serves to confirm the correctness of the response. In
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
385
the case of an error, feedback would provide information which could be used to correct the error. In terms of the M-C model, an error would be the result of a knowledge structure deficit that pertained to either context, class or specific representation. Consequently, informative feedback would be corrective at any stage depending upon the nature of the feedback. Although feedback has been demonstrated to facilitate classroom learning, performance following feedback is typically not perfect. Consequently, a consideration of errors made following feedback would provide insight into the role of informative feedback in meaningful learning. For example, when number of correct responses is the dependent variable under consideration, limited information concerning cognitive processing’ variables can be obtained because there is more than one way to obtain a correct answer with a M-C format. While there are problems in determining the processing involved in answering correctly, an error analysis would permit the determination of cognitive processes in responding incorrectly. It is assumed that a more adequate understanding of M-C performance can be obtained through the analysis of total test performance defined in terms of both correct and incorrect responding. Error Analysis
In a learning situation where informative feedback is used, the paradigm is one of pretest-informative feedback-post-test. With this design, there are five possible combinations of pre/post-test response sequences. Of these combinations, two are the result of correct responding at post-test. In one instance (right-right), feedback served to confirm a correct response made at pretest. In the second case, (wrong-right), an error was made at pretest and feedback provided the basis for correction. The three remaining response sequences are instances of errors at posttest. The first type of error is perseverative in nature and will be referred to as a same error. In this case, an incorrect response made on the initial test was made again on the second test, even though the learner was informed during feedback as to the correct answer. In this situation, the learner perseverates. More specifically, not only was the same item missed on both occasions, but the same incorrect alternative was selected. The second type of error is a different error. In this case, an item is missed on both occasions, but the error is not perseverative in nature. For example, while the same item is missed on both occasions, different incorrect alternatives are selected on each occasion. In the case of the different error, the initial wrong response could be due to either incorrect information or an unlucky guess resulting from a lack of information. Regardless, the distinguishing feature appears to be that at feedback, sufficient information was encoded so that at post-test the learner knows that the initial response (regardless of source) was wrong, but insufficient information was available for accommodation (responding correctly).
386
GARY
D. PHYE
The third type of error is a new error. In this case, an item which was answered correctly on the initial test is missed on the post-test, following informative feedback. This response sequence appears to be the result of a lucky correct guess on the initial test, the ignoring of feedback, then the commission of an error on retest. What may be happening here is the failure of a subject to attend to informative feedback because, “I got it right, therefore I must know the answer.” At this point, it is necessary to reiterate the present conceptualization of informative feedback in terms of its informational consequences. Although motivational consequences are worthy of consideration, they were not considered in the present studies. It was felt that the present focus (i.e., informational consequences) is consistent with the predominate view held by classroom teachers that the utility of information feedback is in terms of its informational consequences that can be used by the learner to improve performance. Purpose
The following studies are initial efforts at testing the proposed M-C processing model, the utility of the error analysis, and the processing of informative feedback. Experiment 1 is concerned with testing aspects of the M-C model. Since the assumption is made that the processing involves the use of retrieval cues, it was hypothesized that both semantic and perceptual retrieval cues could be used to produce correct answers at post-test. In keeping with a depth of processing perspective as it would apply to performance on a retention test with a multiple-choice format (Lockhart, Craik, & Jacoby, 1976), it was hypothesized that length of retention interval would interact with type of retention test. Specifically, performance on the retention test that required the use of semantic retrieval cues should be superior to performance on other types of retention tests at delayed retention. In experiment 2, the nature of immediate feedback defined in terms of different amounts of informational content was investigated. It was hypothesized that a feedback procedure that provided the greatest amount of information would produce the best retention performance at post-test. Since the use of the error analysis is still in its exploratory stages, no specific hypotheses were formulated. EXPERIMENT Method Subjects.
1
Subjects were 180 male and female volunteers regularly enrolled in introductory psychology courses at a large midwestern university. Data were collected in group sessions with 10 subjects per group. Assignment to groups was done randomly. Procedure. A 40-item multiple-choice test with individual items consisting of a stem, correct answer, and two alternatives was used as the pretest. The stem was a short definition; the correct answer and incorrect alternatives were single words (Phye PI al., 1976).
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
387
Informative feedback was provided in a feedback booklet in one of three forms: (1) stem and only correct alternative; (2) stem and designated correct answer with incorrect alternatives from the pretest; or (3) stem and designated correct answer with the two incorrect ahernatives from the pretest plus two previously unseen incorrect alternatives. In all conditions, feedback was read aloud by both experimenter and subjects. Following the completion of the pretest, all subjects were dismissed and asked to return 48 hr later for informative feedback. This delayed feedback interval was selected because it approximates the typical classroom testing-feedback interval in the college classroom. Upon completion of the respective feedback sessions, half of the subjects in every condition received an immediate retention test. The remaining subjects returned 7 days later for the delayed retention test. Retention tests. Retention tests consisted of items having a stem and three alternatives. Retention test items took one of three forms: (a) the pretest (repeated test); (b) the pretest stem and correct answer with new incorrect alternatives not present on the pretest or during feedback (new alternative test); and (c) the form where the pretest correct answer (single term) served as the stem and the pretest stem (definition) as the correct response (term to definition test). On the term to definition test (T-D), definitions were substituted for the single terms that had served as incorrect alternatives on the pretest. For example, if “mellifluous” had been a distractor on pretest, on the retention test the definition “flowing sweetly and smoothly” was substituted. The T-D retention test was used in order to preclude the elimination of incorrect ahernatives on the basis of phonological and/or orthographic cues associated with incorrect alternatives on the initial test or at feedback. Thus, the basis for the discrimination would appear to be semantic. A discrimination made on the basis of semantic encoding would require the use of retrieval cues in order to enter memory store. An alternative to a discrimination process based on the use of semantic retrieval cues would be an associative hookup between the stem and correct answer. Whether the processing is associative or stimulus discrimination based on semantic encoding, the use of retrieval cues would be involved (Lockhart et al., 1976). In experiment I, all three types of errors (same, different, and new) can be made on the repeated (R-T) and the term to definition (T-D) retention tests. Only new and different errors are possible on the new alternatives test (N-A) retention test.
Results Correct responses. Correct responses were analyzed using an analysis
of variance with repeated measures. Repeated measures were pre- and post-test (retention) performance. Components were combined factorially to produce a 3 (form of feedback) by 3 (type of retention test) by 2 (retention interval) by 2 (repeated measures) design. Results indicated significant main effects for type of retention test, F(2,162) = 21.56, p < .Ol, retention interval, F( 1,162) = 20.13, p < .Ol, and repeated measures, F(1,162) = 2133.41,~ < .Ol. These main effects were also containedin two significant interactions; type of retention test X retention interval, F(2, 162) = 6.85, p < .Ol, and type of retention test x repeated testing, F(2, 162) = 13.41, p < .Ol. Newman-Keuls test of means was used to test significant main effects. Means for type of retention test were 27.41 (T-D), 24.39 (N-A), and 23.36 (R-T). All differences were significant (p < .05). Immediate retention (mean = 26.25) was significantly better (p < .05) than delayed retention (mean = 24.27). As reflected in the significant main ef-
388
GARY
D. PHYE
feet for repeated testing, performance following feedback (mean = 33.45) was superior to pretest performance (mean = 17.33). As shown in Fig. 2, the significant interaction of type of retention test by retention interval indicated that delayed retention was best when the test consisted of a single word as the stem and definitions served as alternatives. A Newman- Keuls test of means indicated significant differences (p < .05) between the R-T retention test and the T-D and N-A retention tests on immediate retention. At delayed retention, all retention tests differed significantly. The type of retention test by repeated measures interaction indicated that learners who received the T-D retention test performed better (p < .OS)on the pretest than either the N-A or R-T groups who didn’t differ significantly. However, following feedback, performance on the T-D retention test was still superior to N-A test performance and N-A was superior to R-T test performance, regardless of retention interval. Error Analysis
Types of error were recorded for each subject and the sum of each was expressed as a proportion of total errors. While the proportions would vary across subjects, in all cases where errors were made, they would sum to 1.00 for any individual. Since error data were expressed as proportions, arc sine transformations were performed on the data prior to data analysis (Winer, 1962). Data were analyzed using a multivariate analysis of variance.
-T-D
z
24-
w (L
23-
+ v
22-
k E v
2l20-
f
I
I Immedlole
RETENTION
Delayed
INTERVAL
FIG. 2. Correct responses at immediate and delayed retention of retention tests following delayed feedback.
for each of the three types
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
389
A 2 (type of retention test) by 2 (retention interval) MANOVA with same, different, and new errors as dependent variables was used to analyze data from the R-T and T-D retention tests. The N-A retention test data were analyzed separately since only new and different errors were possible on this test. Significant overall effects existed for type of test, F (3,113) = 3.08, p < .03, and the type of test x retention interval interaction, F(3,113) = 3.29, p < .02. The type of test effect was significant for same errors, F(1,115) = 6.49, p < .Ol. No other differences (i.e., new or different errors) were significant. Significantly more same (perseverative) errors were committed on the R-T retention test (mean = .49) than the T-D retention test (mean = .33). The significant overall interaction type of test by retention interval was further analyzed. Significant differences were observed for new errors, F( 1,115) = 6.27, p < .Ol. There was a significant increase in the commission of new errors on the T-D retention test at delayed retention. Also, more new errors were made on the T-D retention test than on the R-T test at delayed retention or on the T-D and R-T tests at immediate retention. A one-way MANOVA with new and different errors as dependent variables was used to analyze data from the N-A retention tests. No significant overall differences existed between immediate and delayed retention tests. In experiment 1, only delayed feedback was considered. Information concerning immediate feedback is also necessary. Additionally, since type of feedback did not produce a differential effect on post-test performance, a change was made in experiment 2 in the nature of feedback provided. EXPERIMENT
2
Method Subjects. Subjects were 60 volunteers from introductory
psychology courses at a large state university. All subjects received extra credit for participation. Procedure. Only one type of test, the R-T test from experiment 1 was employed for both the pretest and post-test. Immediate feedback was furnished within a few minutes of the completion of the 40-item pretest. The amount of informative feedback was manipulated in three feedback conditions. In all conditions, subjects received minimal informative feedback consisting of their answer sheet from the pretest and a feedback sheet containing the stems and only correct answers. The feedback condition which provided the least amount of information was described above. The second and third feedback conditions were as described above with the following additions. In feedback condition 2, the answer sheet had been scored and the total number of correct responses was noted. In feedback condition 3, the total number of correct and individual items missed were indicated. Following feedback, either an immediate or delayed retention (5 days) test was administered. The research design was a 3 (type of feedback) by 2 (retention interval) factorial. midwestern
Results Correct Response Data. Correct response data were analyzed using a 2
(retention interval)
x
3 (type of feedback)
x
2 (repeated measures)
390
GARY D. PHYE
analysis of variance. Significant main effects were found for retention interval F(1,54) = 4.34, p < .04, and repeated measures, F(1,54) = 472.61, p < .OOl. The within subject main effect for pre/post-test (repeated measures) simply attests to the facilitative effect of informative feedback (pretest mean = 17.23, post-test mean = 31.18). Performance on the immediate retention test (mean = 25.23) was significantly better than on the delayed retention test (mean = 23.18). Significant interactions of retention interval by repeated measures, F (154) = 11.57, p < .Ol, and feedback by repeated measures, F(2,54) = 5.57, p < .Ol, were observed. The group that received feedback plus answer sheet, score correct, and missed items (condition 3) performed better (p < .05) on the pretest than other conditions. However, on posttests, the only significant difference was found between the feedback and answer sheet only condition (condition 1, mean = 32.95) and feedback conditions 2 (mean = 30.25) and 3 (mean = 30.45). The retention interval by pre/post-test interaction reflected the significant main effect of retention interval. While pretest performance did not differ significantly, immediate retention (mean = 33.30) was superior (p < .05) to delayed retention (mean = 29.07). Error analysis. Data were transformed and analyzed using a 2 (retention interval) by 3 (type of feedback) MANOVA with same, different, and new errors as dependent variables. A significant overall main effect was observed for retention effect, F(3,54) = 6.53, p < .OOl. Analysis of the significant overall retention effect indicated significant effects for same error F(1,54) = 5.89, p < ‘02. As shown in Table 1, significantly more same errors were made at delayed retention (mean = .65) than at immediate retention (mean = .35). Significantly more different errors were made at immediate retention (mean = .32) than delayed retention (mean = .18). DISCUSSION
These studies did not include a no-feedback control condition. Such a control would be used to partial out the effects of warm-up, practice, or learning to learn on post-test performance. In the present studies, however, such a control is immaterial. Whether the source of post-test imTABLE 1 TRANSFORMED MEAN PROPORTION OF ERRORS MADE AT IMMEDIATE AND DELAYED RETENTION FOLLOWING IMMEDIATE FEEDBACK
Type of error Retention interval
Same
Different
New
Immediate retention test Delaved retention test
.35 .65
.32 .I8
.36 .24
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
391
provement is the experimental feedback manipulation per se or the feedback manipulation plus warm-up, practice, or learning to learn is not an issue in the present studies. Also, as has been noted by Kulhavy (1977), the facilitating effect of informational feedback on meaningful verbal material is a well established fact. Correct Response Data
As anticipated at the outset, when feedback is delayed and retention is measured using a multiple-choice format, correct response data can be considered within the context of a depth of processing perspective (Lockhart et al., 1976). Additionally, Lockhart et al. have provided the following operational definition of the retrieval process: “Such a definition identifies retrieval effects with performance which occurs as a function of conditions at time of test, all conditions prior to the test phase having been kept constant. This definition is satisfied by paradigms such as those which vary the context of recognition test items, or the availability of cues for recall” (p. 75). Two post-tests in experiment 1 met the aforementioned condition where a retrieval process would be used. This was the case for the N-A and T-D retention tests, although the N-A and T-D retention tests required the use of different types of retrieval cues. On the R-T retention test, which was the same as the pretest, the operational definition of a test situation where a retrieval process would be used, was not met. When the use of a retrieval process was required by the nature of the retention test, retention performance following delayed feedback was superior to retention performance on a retention test (R-T) where the use of a retrieval process was not required. More specifically, data reported in Fig. 2 indicate that retention performance on a retention test that doesn’t require the use of a retrieval process is significantly (p < .05) poorer at both immediate and delayed retention. Further, the type of retrieval cue used makes a difference in retention performance. At immediate retention, no differences existed in performance when either perceptual or semantic retrieval cues were used. However, at delayed retention (5 days) following delayed informative feedback (48 hr); the retrieval process that required the deeper level of processing (i.e., semantic) promoted significantly better retention performance. In fact, there was no drop-off in amount retained between the immediate and delayed retention groups using semantic retrieval cues. In view of the available perceptual retrieval cues at post-test on the N-A retention test, a verbal frequency could account for superior post-test performance following feedback. Evidently, when new alternatives are used as distracters at post-test, verbal discriminations predicated on the use of perceptual retrieval cues (i.e., orthographic and/or phonological cues) can serve as the basis for the retrieval of a verbal frequency differential The second type of retrieval cue available at post-test would be
392
GARY
D.
PHYE
semantic cues that would be used as the basis for the retrieval process on the T-D retention test. Whether the semantic retrieval cues served as the basis for associative processing or a discrimination process predicated on semantic encoding is not addressed in the present studies. In summary, when retrieval processes were used, retention performance was better than when the retention test was identical to the pretest. As is discussed at greater length in the consideration of the error analysis, the nature of the error that appears to account for the poorer performance on the retention test where a retrieval process was not used was the disproportionate commission of perseverative errors. Informative
Feedback
Informative feedback plays an important role in the correction of initial wrong responses (accommodation). While informative feedback is here considered to be simply additional information without special properties, the amount and timing of feedback influences its effectiveness. For instance, in experiment 1, where feedback was delayed, feedback in the form of correct answer or correct answer plus alternatives did not differentially effect post-test performance. However, in experiment 2, where feedback was immediate and amount of feedback was varied, feedback in the form of only correct answer plus the answer sheet from the pretest was superior to other forms of feedback. The feedback condition thought to provide the least information produced the greatest improvement on the post-test. This would suggest the possibility of a threshold effect in terms of the amount of information provided by the feedback procedure. A threshold hypothesis would posit that when more than sufficient information needed to correct or confirm an answer is provided, it does not have a facilitative effect upon the learner’s ability to use the feedback. The consideration here is what is meant by sufficient information. When the answer sheet was present during immediate feedback (experiment 2), the learner was not required to remember his/her answer on the post-test as was the case in the first study. Also, when the answer sheet plus feedback was provided, a careful monitoring of the information could provide the same amount of information that was provided by the experimenter in feedback conditions 2 and 3. Thus, a close monitoring of answer sheet and feedback may have made the inclusion of additional information pertaining to correct total and/or items missed, superfluous, and distracting. Whether the nature of information found to be most effective at immediate feedback would also be most effective at delayed feedback cannot be answered by the present studies. Error Analysis
The error analysis provides the basis for a consideration of the nature of the processing involved in the failure to use informative feedback to serve
THE PROCESSING
OF INFORMATIVE
FEEDBACK
393
a corrective function. Following immediate feedback, differences exist at immediate and delayed retention in the types of processing that are responsible for a failure to accommodate (i.e., correct pretest errors following feedback). For example, the occurrence of perseverative (same) errors was more frequent on delayed than immediate retention tests. These errors are the result of proactive interference which could be any combination of orthographic, phonological, or semantic cues. Further evidence that proactive interference accounts for the commission of perseverative errors is provided by the data from the T-D retention test in experiment 1. When the retention test required the use of retrieval cues that are primarily semantic in nature, and the possibility of proactive interference from orthographic and/or phonological cues is minimized, the incidence of perseverative errors decreased. Data from experiment 2, where feedback was immediate and pretest and post-test were identical, also speak to this point. Again, there was a significant effect for perseverative error on the delayed retention test. When pretest and post-test measures are identical and the retention test is delayed, perseverative errors account for a significant proportion of the failures to accommodate following feedback in multiple-choice performance. In summary, present results demonstrate the feasibility of considering both correct responses and errors when analyzing M-C test performance. Also, data demonstrating the use of retrieval processes must be considered as providing support for Phye’s (1977) model of multiple-choice processing. While further testing is required, present data attest to the model’s heuristic value. Feedback data suggest a threshold phenomenon as it relates to the amount of information provided at immediate feedback. Feedback containing surplus information doesn’t improve the corrective function of feedback. Additionally, in those cases where feedback served neither a corrective nor confirmatory function (i.e., errors at post-test), the error analysis revealed a disproportionate number of perseverative errors. At the beginning of the article, reference was made to classroom learning. While the specification of processing detail which is the concern of cognitive psychology may have appeared to leave the realm of practice, I must insist that this is not the case. The procedures (groups, paper-pencil tests, etc.) used in the present studies very accurately represent educational and testing contexts. Also, the practical utility of the error analysis model has been demonstrated (Engel & Phye, in press; Phye & Engel, Note 2) by its use to identify individual differences in learner expectations for young and elderly adults in simulated classroom settings. REFERENCES ADAMS, J. A. LeartGg & mernor~: An infroducriotr. Homewood, Ill.: Dorsey, 1976. ANDERSON, J. R., & BOWER, G. H. Humat~ nssociarir,r memory. Washington, D.C.:
Winston,
1973.
394
GARY
D.
PHYE
ANDERSON, R. C. How to construct achievement tests to assess comprehension. R~L’I’~I~. of Educational RescJarch, 1972, 42, 145- 170. CARROLL, J. B. Promoting language skills: The role of instruction. In D. Klahr (Ed.), c‘ognition and instruction. Hillsdale, N.J.: Halsted Press, 1976. ENGEL, J. B., & PHYE, G. D. Learner expectations and the processing of informative feedback by young and elderly adults. Journal ofEduc~ationa/ Grrontology, in press. KLAHR, D. Cognition and instruction. Hillsdale, New Jersey: Halsted Press, 1976. KULHAVY, R. Feedback in written instruction. Rrvk~ of‘Educ.utiona/ Research. 1977, 47,
211-232. & ANDERSON, R. C. Delayed retention effect with multiple-choice tests. Psychology, 1972, 63, 505-5 12. R. S., CRAIK, F. 1. M., & JACOBY, L. Depth of processing, recognition and recall. In J. Brown (Ed.), Recu// and rrcognition. London/New York: Wiley, 1976. PHYE, G., & BALLER, W. Verbal retention as a function of the informativeness and delay of informative feedback: A replication. Journal of Edrrcutional Psychology. 1970, 61, KULHAVY, Journal LOCKHART,
R.,
of Educational
381-381. PHYE, G., GUGLIEMELLA,J., & SOLA, J. Effects ofdelayed retention on multiple-choice test performance. Contrmporury Educational Psychology. 1976, 1, 26-36. SASSENRATH,J. M. Theory and results on feedback and retention. Journul of Educational Psyc~hology,
1975, 67, 894-899.
SASSENRATH,J. J., & GARVERICK,C. M. Effects of differential feedback from examinations Psychology, 1965, 56, 259-263. on retention and transfer. Journal oJ’Educ~utionu/ STERNBERG,S. High-speed scanning in human memory. Scie~lce. 1966, 1.53,652-662. STERNBERG,S. Memory-scanning: Mental processes revealed by reaction time experiments. American
Sckntist.
1969, 57, 42 I-457.
SUBER, J. R., & ANDERSON,R. C. Delay-retention effect in natural classroom settings. Journal of Educational Psyc,ho/ogy, 1975, 67, 170- 173. TULVING, E. Episodic and semantic memory. In E. Tulving & W. Donaldson (Eds.), Organization of memory. New York: Academic Press, 1972. WINER, B. J. Stutisric~alprinciples in cxpc,rimerltu/ design. New York: McGraw-Hill, 1962. WITTROCK,M. C. The cognitive movement in instruction. Educutiona/ Psycho/ogist, 1978, 13, 15-29.
REFERENCE
NOTES
D. Thr role of informativc~jti~dbuck in productive, Icurrzing. Paper presented at the annual meeting of the American Educational Research Association, New York, 1977. 2. PHYE, G. D., & ENGEL, J. B. The udult Irurner: Learning to learn in an ucudcmic sptting. Manuscript submitted for publication, 1979. 1. PHYE, G.
EDUCATIONAL
4, 381-394 (1979)
PSYCHOLOGY
The Processing of Informative Feedback Multiple-Choice Test Performance
about
GARY D. PHYE Iotzw
Stare
Univcmity
From a cognitive perspective, informative feedback is assumed to be confirmatory or corrective. However, feedback is not always effective and errors persist. In cases where errors persist, an analysis of errors as well as correct responses is recommended in order to better understand the processing of informative feedback. A model of multiple-choice processing is introduced to provide the context for a consideration of the role of error analysis. Data from experiment 1 demonstrate the use of semantic retrieval cues in multiple-choice performance. While informative feedback improves post-test performance, it is difficult to demonstrate a differential effect for type of feedback. Experiment 2 suggests a threshold effect for informative feedback. Information beyond that which is necessary for the confirmation of a correct response and/or the correction of an error has a deleterious effect on post-test performance. Error analysis indicated a disproportionate number of perseverative errors when the use of a retrieval process was not required at post-test.
Since the early work of E. L. Thorndike, it has been assumed that informative feedback in the form of knowledge of results facilitates performance. Whether the improvement in performance is the result of motivational or informational consequences depends in part on the type of task under consideration. When the task is verbal and meets the definition of meaningful learning frequently seen in the classroom, the informational consequences appear to predominate. For example, Kulhavy and Anderson (1972), Phye, Gugliemella, and Sola (1976), and Sassenrath (1975) have marshaled evidence for the role of informative feedback in the facilitation of classroom learning through the correction of previous errors. In the classroom, a multiple-choice (M-C) format is frequently used to test abilities, skills, and knowledge as attested to by its extensive use on teacher-made tests as well as standardized achievement and aptitude tests. In the experimental literature, the M-C format has been used to measure the comprehension and retention of classroom materials following informative feedback (Phye & Baller, 1970; Sassenrath & Garverick, 1965; Suber & Anderson, 1975). Also, Anderson (1972) has recommended the use of a multiple-choice format with paraphrasing in order to measure Based on the paper “The role of informative feedback in productive learning” presented at the American Educational Research Association annual meeting, New York, 1977. Appreciation is extended to Joseph Gugliemella for his help with the data collection in experiment 1. 381 0361-476X/79/040381-14$02.00/0 Copyright Q 1979 by Academic Press, Inc. All rights of reproduction in any form reserved.
382
GARY
D. PHYE
the comprehension of meaningful material. Meaningful learning is here considered to be semantic in nature. Specifically, semantic memory (Tulving, 1972) is the organized knowledge a person possesses about words, meanings, relations, concepts, rules, etc. Input into semantic memory is processed and related to existing cognitive structures. In addition to defining the type of learning to be studied and the type of retention measure used, a conceptualization of the nature and role of informative feedback is necessary. In keeping with the semantic nature of the present definition of meaningful learning, the nature of informative feedback is herein considered in terms of its semantic content. Consequently, feedback could take the form of words, rules, concepts, meanings, etc. Hence, the role of feedback is either to provide confirmation of existing knowledge structures and schemes or to provide the semantic basis for the modification of an existing structure or scheme. Having specified a M-C format as the retention measure, consideration will be directed toward the development of a theoretical model of multiple-choice performance from a cognitive perspective. The M-C model is presented in order to provide a context for a consideration of errors following informative feedback. The error analysis that will be proposed provides the basis for identifying breakdowns in the processing of informative feedback. In light of the recent emphasis on the investigation of cognitive processes involved in instructional situations (Klahr, 1976), and an increased utilization of information processing theory, an examination of theoretical assumptions is essential when examining the role of informative feedback in instruction. Cognitive Perspective “A cognitive approach implies that learning from instruction is scientifically more productively studied as an internally, cognitively mediated process than as a direct product of the environment, people, or factors external to the learner” (Wittrock, 1978, p. 15). This view is consistent with that expressed by Carroll (1976). Cognitive theory would assert that “information is what is learned and would object to behavioral theory’s postulation that it is the responses that are learned.” “According to cognitive theory, learning to make particular responses is an internal process, as is also the decision to emit them on a particular occasion and under particular conditions.” A corollary assumption is that “feedback and correction, in cognitive theory, would be regarded as merely one kind of information contributing to learning-though frequently an important kind of information” (Carroll, 1976, p. 17). Also, there have been recent modifications in the theoretical conceptualization of the recognition process. Adams (1976) has pointed out that, while it is apparent that we search our memory at recall, it is not readily
THE PROCESSING
OF INFORMATIVE
FEEDBACK
383
apparent that search and retrieval operate for recognition. Clarification on this point has been offered by Sternberg (1966, 1969) who presents evidence for the consideration of search and retrieval as factors in recognition. Also, within the context of Human Associate Memory (HAM), Anderson and Bower (1973) consider the recognition of classes, individuals, and familiar patterns (perceptual concepts) with or without retrieval of context. These considerations of the processing involved in the recognition process would suggest a reconceptualization of multiple-choice test performance. Multiple-Choice Test Performance In order to consider the role of informative feedback when learning is assessedusing a M-C format, a M-C processing model has been proposed (Phye, Note 1). As indicated in Fig. 1, the model incorporates the cognitive assumption that instances of both storage and retrieval from memory are included in M-C performance. The model assumes several stages of processing and involves more than a simple single-stage discrimination process. A major reason for the inability to account for M-C performance without the consideration of a retrieval process is that the stem of a test item may serve to direct attention or serve as a retrieval cue for the establishment of a semantic context. For instance, when initially encountering a multiple-choice test item, a learner will typically read the stem followed by a reading of the alternatives. A choice without retrieval of context may be possible if the stem furnishes grammatical or syntactical cues that can be used in selecting an alternative. If this situation occurs, the subject goes to the next item. If not, the subject will likely use the stem as a retrieval cue in order to establish the context for memory search. If the stem has not served as a cue for the retrieval of context, the subject will guess at this point and go to the next item. In addition to the use of the stem as a retrieval cue in order to establish context, the stem may also serve as a cue for the retrieval of classes of information within a context. For example, if the stem dealt with classroom management and the promotion of consistent responding, the context would be operant conditioning principles and the class would be schedules of reinforcement. The assumption is made that the retrieval of context is a necessary but not sufficient condition for class retrieval. If the item stem can be used to retrieve both context and a class of information, the subject can focus on alternatives. Having previously narrowed memory search to a class of information, that information can be used to eliminate alternatives that fall outside the class definition. The basis for the elimination of alternatives would be the observed incongruity between the memory schema defined in terms of context and class and the alternatives contained in the
384
GARY D. PHYE
No-
-YSt FIG. 1. Processing model of multiple-choice test performance.
item. A subject may stop at this point and select an alternative. However, the subject may persevere and engage in one more stage of processing. Having narrowed the search to a class within a context, a specific item or instance from within a class may be retrieved and serve as the basis for a match between a specific memory representation and remaining alternatives. In some cases, a subject need not go through all of the stages in a serial manner. There are instances where it must be assumed that a person could bypass the class search and go directly from context to a specific memory representation. This situation would most likely exist when the item was very meaningful. The present model would suggest that the high level of correct response performance typically associated with M-C retention measures is not due to the simplicity (i.e., storage but not retrieval) of the required processing. Rather, the high level of performance appears to be due to the abundance of retrieval cues and their use in selecting an alternative. Informative
Feedback and M-C Performance
Informative feedback can serve two functions. In the case of a correct answer, feedback serves to confirm the correctness of the response. In
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
385
the case of an error, feedback would provide information which could be used to correct the error. In terms of the M-C model, an error would be the result of a knowledge structure deficit that pertained to either context, class or specific representation. Consequently, informative feedback would be corrective at any stage depending upon the nature of the feedback. Although feedback has been demonstrated to facilitate classroom learning, performance following feedback is typically not perfect. Consequently, a consideration of errors made following feedback would provide insight into the role of informative feedback in meaningful learning. For example, when number of correct responses is the dependent variable under consideration, limited information concerning cognitive processing’ variables can be obtained because there is more than one way to obtain a correct answer with a M-C format. While there are problems in determining the processing involved in answering correctly, an error analysis would permit the determination of cognitive processes in responding incorrectly. It is assumed that a more adequate understanding of M-C performance can be obtained through the analysis of total test performance defined in terms of both correct and incorrect responding. Error Analysis
In a learning situation where informative feedback is used, the paradigm is one of pretest-informative feedback-post-test. With this design, there are five possible combinations of pre/post-test response sequences. Of these combinations, two are the result of correct responding at post-test. In one instance (right-right), feedback served to confirm a correct response made at pretest. In the second case, (wrong-right), an error was made at pretest and feedback provided the basis for correction. The three remaining response sequences are instances of errors at posttest. The first type of error is perseverative in nature and will be referred to as a same error. In this case, an incorrect response made on the initial test was made again on the second test, even though the learner was informed during feedback as to the correct answer. In this situation, the learner perseverates. More specifically, not only was the same item missed on both occasions, but the same incorrect alternative was selected. The second type of error is a different error. In this case, an item is missed on both occasions, but the error is not perseverative in nature. For example, while the same item is missed on both occasions, different incorrect alternatives are selected on each occasion. In the case of the different error, the initial wrong response could be due to either incorrect information or an unlucky guess resulting from a lack of information. Regardless, the distinguishing feature appears to be that at feedback, sufficient information was encoded so that at post-test the learner knows that the initial response (regardless of source) was wrong, but insufficient information was available for accommodation (responding correctly).
386
GARY
D. PHYE
The third type of error is a new error. In this case, an item which was answered correctly on the initial test is missed on the post-test, following informative feedback. This response sequence appears to be the result of a lucky correct guess on the initial test, the ignoring of feedback, then the commission of an error on retest. What may be happening here is the failure of a subject to attend to informative feedback because, “I got it right, therefore I must know the answer.” At this point, it is necessary to reiterate the present conceptualization of informative feedback in terms of its informational consequences. Although motivational consequences are worthy of consideration, they were not considered in the present studies. It was felt that the present focus (i.e., informational consequences) is consistent with the predominate view held by classroom teachers that the utility of information feedback is in terms of its informational consequences that can be used by the learner to improve performance. Purpose
The following studies are initial efforts at testing the proposed M-C processing model, the utility of the error analysis, and the processing of informative feedback. Experiment 1 is concerned with testing aspects of the M-C model. Since the assumption is made that the processing involves the use of retrieval cues, it was hypothesized that both semantic and perceptual retrieval cues could be used to produce correct answers at post-test. In keeping with a depth of processing perspective as it would apply to performance on a retention test with a multiple-choice format (Lockhart, Craik, & Jacoby, 1976), it was hypothesized that length of retention interval would interact with type of retention test. Specifically, performance on the retention test that required the use of semantic retrieval cues should be superior to performance on other types of retention tests at delayed retention. In experiment 2, the nature of immediate feedback defined in terms of different amounts of informational content was investigated. It was hypothesized that a feedback procedure that provided the greatest amount of information would produce the best retention performance at post-test. Since the use of the error analysis is still in its exploratory stages, no specific hypotheses were formulated. EXPERIMENT Method Subjects.
1
Subjects were 180 male and female volunteers regularly enrolled in introductory psychology courses at a large midwestern university. Data were collected in group sessions with 10 subjects per group. Assignment to groups was done randomly. Procedure. A 40-item multiple-choice test with individual items consisting of a stem, correct answer, and two alternatives was used as the pretest. The stem was a short definition; the correct answer and incorrect alternatives were single words (Phye PI al., 1976).
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
387
Informative feedback was provided in a feedback booklet in one of three forms: (1) stem and only correct alternative; (2) stem and designated correct answer with incorrect alternatives from the pretest; or (3) stem and designated correct answer with the two incorrect ahernatives from the pretest plus two previously unseen incorrect alternatives. In all conditions, feedback was read aloud by both experimenter and subjects. Following the completion of the pretest, all subjects were dismissed and asked to return 48 hr later for informative feedback. This delayed feedback interval was selected because it approximates the typical classroom testing-feedback interval in the college classroom. Upon completion of the respective feedback sessions, half of the subjects in every condition received an immediate retention test. The remaining subjects returned 7 days later for the delayed retention test. Retention tests. Retention tests consisted of items having a stem and three alternatives. Retention test items took one of three forms: (a) the pretest (repeated test); (b) the pretest stem and correct answer with new incorrect alternatives not present on the pretest or during feedback (new alternative test); and (c) the form where the pretest correct answer (single term) served as the stem and the pretest stem (definition) as the correct response (term to definition test). On the term to definition test (T-D), definitions were substituted for the single terms that had served as incorrect alternatives on the pretest. For example, if “mellifluous” had been a distractor on pretest, on the retention test the definition “flowing sweetly and smoothly” was substituted. The T-D retention test was used in order to preclude the elimination of incorrect ahernatives on the basis of phonological and/or orthographic cues associated with incorrect alternatives on the initial test or at feedback. Thus, the basis for the discrimination would appear to be semantic. A discrimination made on the basis of semantic encoding would require the use of retrieval cues in order to enter memory store. An alternative to a discrimination process based on the use of semantic retrieval cues would be an associative hookup between the stem and correct answer. Whether the processing is associative or stimulus discrimination based on semantic encoding, the use of retrieval cues would be involved (Lockhart et al., 1976). In experiment I, all three types of errors (same, different, and new) can be made on the repeated (R-T) and the term to definition (T-D) retention tests. Only new and different errors are possible on the new alternatives test (N-A) retention test.
Results Correct responses. Correct responses were analyzed using an analysis
of variance with repeated measures. Repeated measures were pre- and post-test (retention) performance. Components were combined factorially to produce a 3 (form of feedback) by 3 (type of retention test) by 2 (retention interval) by 2 (repeated measures) design. Results indicated significant main effects for type of retention test, F(2,162) = 21.56, p < .Ol, retention interval, F( 1,162) = 20.13, p < .Ol, and repeated measures, F(1,162) = 2133.41,~ < .Ol. These main effects were also containedin two significant interactions; type of retention test X retention interval, F(2, 162) = 6.85, p < .Ol, and type of retention test x repeated testing, F(2, 162) = 13.41, p < .Ol. Newman-Keuls test of means was used to test significant main effects. Means for type of retention test were 27.41 (T-D), 24.39 (N-A), and 23.36 (R-T). All differences were significant (p < .05). Immediate retention (mean = 26.25) was significantly better (p < .05) than delayed retention (mean = 24.27). As reflected in the significant main ef-
388
GARY
D. PHYE
feet for repeated testing, performance following feedback (mean = 33.45) was superior to pretest performance (mean = 17.33). As shown in Fig. 2, the significant interaction of type of retention test by retention interval indicated that delayed retention was best when the test consisted of a single word as the stem and definitions served as alternatives. A Newman- Keuls test of means indicated significant differences (p < .05) between the R-T retention test and the T-D and N-A retention tests on immediate retention. At delayed retention, all retention tests differed significantly. The type of retention test by repeated measures interaction indicated that learners who received the T-D retention test performed better (p < .OS)on the pretest than either the N-A or R-T groups who didn’t differ significantly. However, following feedback, performance on the T-D retention test was still superior to N-A test performance and N-A was superior to R-T test performance, regardless of retention interval. Error Analysis
Types of error were recorded for each subject and the sum of each was expressed as a proportion of total errors. While the proportions would vary across subjects, in all cases where errors were made, they would sum to 1.00 for any individual. Since error data were expressed as proportions, arc sine transformations were performed on the data prior to data analysis (Winer, 1962). Data were analyzed using a multivariate analysis of variance.
-T-D
z
24-
w (L
23-
+ v
22-
k E v
2l20-
f
I
I Immedlole
RETENTION
Delayed
INTERVAL
FIG. 2. Correct responses at immediate and delayed retention of retention tests following delayed feedback.
for each of the three types
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
389
A 2 (type of retention test) by 2 (retention interval) MANOVA with same, different, and new errors as dependent variables was used to analyze data from the R-T and T-D retention tests. The N-A retention test data were analyzed separately since only new and different errors were possible on this test. Significant overall effects existed for type of test, F (3,113) = 3.08, p < .03, and the type of test x retention interval interaction, F(3,113) = 3.29, p < .02. The type of test effect was significant for same errors, F(1,115) = 6.49, p < .Ol. No other differences (i.e., new or different errors) were significant. Significantly more same (perseverative) errors were committed on the R-T retention test (mean = .49) than the T-D retention test (mean = .33). The significant overall interaction type of test by retention interval was further analyzed. Significant differences were observed for new errors, F( 1,115) = 6.27, p < .Ol. There was a significant increase in the commission of new errors on the T-D retention test at delayed retention. Also, more new errors were made on the T-D retention test than on the R-T test at delayed retention or on the T-D and R-T tests at immediate retention. A one-way MANOVA with new and different errors as dependent variables was used to analyze data from the N-A retention tests. No significant overall differences existed between immediate and delayed retention tests. In experiment 1, only delayed feedback was considered. Information concerning immediate feedback is also necessary. Additionally, since type of feedback did not produce a differential effect on post-test performance, a change was made in experiment 2 in the nature of feedback provided. EXPERIMENT
2
Method Subjects. Subjects were 60 volunteers from introductory
psychology courses at a large state university. All subjects received extra credit for participation. Procedure. Only one type of test, the R-T test from experiment 1 was employed for both the pretest and post-test. Immediate feedback was furnished within a few minutes of the completion of the 40-item pretest. The amount of informative feedback was manipulated in three feedback conditions. In all conditions, subjects received minimal informative feedback consisting of their answer sheet from the pretest and a feedback sheet containing the stems and only correct answers. The feedback condition which provided the least amount of information was described above. The second and third feedback conditions were as described above with the following additions. In feedback condition 2, the answer sheet had been scored and the total number of correct responses was noted. In feedback condition 3, the total number of correct and individual items missed were indicated. Following feedback, either an immediate or delayed retention (5 days) test was administered. The research design was a 3 (type of feedback) by 2 (retention interval) factorial. midwestern
Results Correct Response Data. Correct response data were analyzed using a 2
(retention interval)
x
3 (type of feedback)
x
2 (repeated measures)
390
GARY D. PHYE
analysis of variance. Significant main effects were found for retention interval F(1,54) = 4.34, p < .04, and repeated measures, F(1,54) = 472.61, p < .OOl. The within subject main effect for pre/post-test (repeated measures) simply attests to the facilitative effect of informative feedback (pretest mean = 17.23, post-test mean = 31.18). Performance on the immediate retention test (mean = 25.23) was significantly better than on the delayed retention test (mean = 23.18). Significant interactions of retention interval by repeated measures, F (154) = 11.57, p < .Ol, and feedback by repeated measures, F(2,54) = 5.57, p < .Ol, were observed. The group that received feedback plus answer sheet, score correct, and missed items (condition 3) performed better (p < .05) on the pretest than other conditions. However, on posttests, the only significant difference was found between the feedback and answer sheet only condition (condition 1, mean = 32.95) and feedback conditions 2 (mean = 30.25) and 3 (mean = 30.45). The retention interval by pre/post-test interaction reflected the significant main effect of retention interval. While pretest performance did not differ significantly, immediate retention (mean = 33.30) was superior (p < .05) to delayed retention (mean = 29.07). Error analysis. Data were transformed and analyzed using a 2 (retention interval) by 3 (type of feedback) MANOVA with same, different, and new errors as dependent variables. A significant overall main effect was observed for retention effect, F(3,54) = 6.53, p < .OOl. Analysis of the significant overall retention effect indicated significant effects for same error F(1,54) = 5.89, p < ‘02. As shown in Table 1, significantly more same errors were made at delayed retention (mean = .65) than at immediate retention (mean = .35). Significantly more different errors were made at immediate retention (mean = .32) than delayed retention (mean = .18). DISCUSSION
These studies did not include a no-feedback control condition. Such a control would be used to partial out the effects of warm-up, practice, or learning to learn on post-test performance. In the present studies, however, such a control is immaterial. Whether the source of post-test imTABLE 1 TRANSFORMED MEAN PROPORTION OF ERRORS MADE AT IMMEDIATE AND DELAYED RETENTION FOLLOWING IMMEDIATE FEEDBACK
Type of error Retention interval
Same
Different
New
Immediate retention test Delaved retention test
.35 .65
.32 .I8
.36 .24
THE
PROCESSING
OF INFORMATIVE
FEEDBACK
391
provement is the experimental feedback manipulation per se or the feedback manipulation plus warm-up, practice, or learning to learn is not an issue in the present studies. Also, as has been noted by Kulhavy (1977), the facilitating effect of informational feedback on meaningful verbal material is a well established fact. Correct Response Data
As anticipated at the outset, when feedback is delayed and retention is measured using a multiple-choice format, correct response data can be considered within the context of a depth of processing perspective (Lockhart et al., 1976). Additionally, Lockhart et al. have provided the following operational definition of the retrieval process: “Such a definition identifies retrieval effects with performance which occurs as a function of conditions at time of test, all conditions prior to the test phase having been kept constant. This definition is satisfied by paradigms such as those which vary the context of recognition test items, or the availability of cues for recall” (p. 75). Two post-tests in experiment 1 met the aforementioned condition where a retrieval process would be used. This was the case for the N-A and T-D retention tests, although the N-A and T-D retention tests required the use of different types of retrieval cues. On the R-T retention test, which was the same as the pretest, the operational definition of a test situation where a retrieval process would be used, was not met. When the use of a retrieval process was required by the nature of the retention test, retention performance following delayed feedback was superior to retention performance on a retention test (R-T) where the use of a retrieval process was not required. More specifically, data reported in Fig. 2 indicate that retention performance on a retention test that doesn’t require the use of a retrieval process is significantly (p < .05) poorer at both immediate and delayed retention. Further, the type of retrieval cue used makes a difference in retention performance. At immediate retention, no differences existed in performance when either perceptual or semantic retrieval cues were used. However, at delayed retention (5 days) following delayed informative feedback (48 hr); the retrieval process that required the deeper level of processing (i.e., semantic) promoted significantly better retention performance. In fact, there was no drop-off in amount retained between the immediate and delayed retention groups using semantic retrieval cues. In view of the available perceptual retrieval cues at post-test on the N-A retention test, a verbal frequency could account for superior post-test performance following feedback. Evidently, when new alternatives are used as distracters at post-test, verbal discriminations predicated on the use of perceptual retrieval cues (i.e., orthographic and/or phonological cues) can serve as the basis for the retrieval of a verbal frequency differential The second type of retrieval cue available at post-test would be
392
GARY
D.
PHYE
semantic cues that would be used as the basis for the retrieval process on the T-D retention test. Whether the semantic retrieval cues served as the basis for associative processing or a discrimination process predicated on semantic encoding is not addressed in the present studies. In summary, when retrieval processes were used, retention performance was better than when the retention test was identical to the pretest. As is discussed at greater length in the consideration of the error analysis, the nature of the error that appears to account for the poorer performance on the retention test where a retrieval process was not used was the disproportionate commission of perseverative errors. Informative
Feedback
Informative feedback plays an important role in the correction of initial wrong responses (accommodation). While informative feedback is here considered to be simply additional information without special properties, the amount and timing of feedback influences its effectiveness. For instance, in experiment 1, where feedback was delayed, feedback in the form of correct answer or correct answer plus alternatives did not differentially effect post-test performance. However, in experiment 2, where feedback was immediate and amount of feedback was varied, feedback in the form of only correct answer plus the answer sheet from the pretest was superior to other forms of feedback. The feedback condition thought to provide the least information produced the greatest improvement on the post-test. This would suggest the possibility of a threshold effect in terms of the amount of information provided by the feedback procedure. A threshold hypothesis would posit that when more than sufficient information needed to correct or confirm an answer is provided, it does not have a facilitative effect upon the learner’s ability to use the feedback. The consideration here is what is meant by sufficient information. When the answer sheet was present during immediate feedback (experiment 2), the learner was not required to remember his/her answer on the post-test as was the case in the first study. Also, when the answer sheet plus feedback was provided, a careful monitoring of the information could provide the same amount of information that was provided by the experimenter in feedback conditions 2 and 3. Thus, a close monitoring of answer sheet and feedback may have made the inclusion of additional information pertaining to correct total and/or items missed, superfluous, and distracting. Whether the nature of information found to be most effective at immediate feedback would also be most effective at delayed feedback cannot be answered by the present studies. Error Analysis
The error analysis provides the basis for a consideration of the nature of the processing involved in the failure to use informative feedback to serve
THE PROCESSING
OF INFORMATIVE
FEEDBACK
393
a corrective function. Following immediate feedback, differences exist at immediate and delayed retention in the types of processing that are responsible for a failure to accommodate (i.e., correct pretest errors following feedback). For example, the occurrence of perseverative (same) errors was more frequent on delayed than immediate retention tests. These errors are the result of proactive interference which could be any combination of orthographic, phonological, or semantic cues. Further evidence that proactive interference accounts for the commission of perseverative errors is provided by the data from the T-D retention test in experiment 1. When the retention test required the use of retrieval cues that are primarily semantic in nature, and the possibility of proactive interference from orthographic and/or phonological cues is minimized, the incidence of perseverative errors decreased. Data from experiment 2, where feedback was immediate and pretest and post-test were identical, also speak to this point. Again, there was a significant effect for perseverative error on the delayed retention test. When pretest and post-test measures are identical and the retention test is delayed, perseverative errors account for a significant proportion of the failures to accommodate following feedback in multiple-choice performance. In summary, present results demonstrate the feasibility of considering both correct responses and errors when analyzing M-C test performance. Also, data demonstrating the use of retrieval processes must be considered as providing support for Phye’s (1977) model of multiple-choice processing. While further testing is required, present data attest to the model’s heuristic value. Feedback data suggest a threshold phenomenon as it relates to the amount of information provided at immediate feedback. Feedback containing surplus information doesn’t improve the corrective function of feedback. Additionally, in those cases where feedback served neither a corrective nor confirmatory function (i.e., errors at post-test), the error analysis revealed a disproportionate number of perseverative errors. At the beginning of the article, reference was made to classroom learning. While the specification of processing detail which is the concern of cognitive psychology may have appeared to leave the realm of practice, I must insist that this is not the case. The procedures (groups, paper-pencil tests, etc.) used in the present studies very accurately represent educational and testing contexts. Also, the practical utility of the error analysis model has been demonstrated (Engel & Phye, in press; Phye & Engel, Note 2) by its use to identify individual differences in learner expectations for young and elderly adults in simulated classroom settings. REFERENCES ADAMS, J. A. LeartGg & mernor~: An infroducriotr. Homewood, Ill.: Dorsey, 1976. ANDERSON, J. R., & BOWER, G. H. Humat~ nssociarir,r memory. Washington, D.C.:
Winston,
1973.
394
GARY
D.
PHYE
ANDERSON, R. C. How to construct achievement tests to assess comprehension. R~L’I’~I~. of Educational RescJarch, 1972, 42, 145- 170. CARROLL, J. B. Promoting language skills: The role of instruction. In D. Klahr (Ed.), c‘ognition and instruction. Hillsdale, N.J.: Halsted Press, 1976. ENGEL, J. B., & PHYE, G. D. Learner expectations and the processing of informative feedback by young and elderly adults. Journal ofEduc~ationa/ Grrontology, in press. KLAHR, D. Cognition and instruction. Hillsdale, New Jersey: Halsted Press, 1976. KULHAVY, R. Feedback in written instruction. Rrvk~ of‘Educ.utiona/ Research. 1977, 47,
211-232. & ANDERSON, R. C. Delayed retention effect with multiple-choice tests. Psychology, 1972, 63, 505-5 12. R. S., CRAIK, F. 1. M., & JACOBY, L. Depth of processing, recognition and recall. In J. Brown (Ed.), Recu// and rrcognition. London/New York: Wiley, 1976. PHYE, G., & BALLER, W. Verbal retention as a function of the informativeness and delay of informative feedback: A replication. Journal of Edrrcutional Psychology. 1970, 61, KULHAVY, Journal LOCKHART,
R.,
of Educational
381-381. PHYE, G., GUGLIEMELLA,J., & SOLA, J. Effects ofdelayed retention on multiple-choice test performance. Contrmporury Educational Psychology. 1976, 1, 26-36. SASSENRATH,J. M. Theory and results on feedback and retention. Journul of Educational Psyc~hology,
1975, 67, 894-899.
SASSENRATH,J. J., & GARVERICK,C. M. Effects of differential feedback from examinations Psychology, 1965, 56, 259-263. on retention and transfer. Journal oJ’Educ~utionu/ STERNBERG,S. High-speed scanning in human memory. Scie~lce. 1966, 1.53,652-662. STERNBERG,S. Memory-scanning: Mental processes revealed by reaction time experiments. American
Sckntist.
1969, 57, 42 I-457.
SUBER, J. R., & ANDERSON,R. C. Delay-retention effect in natural classroom settings. Journal of Educational Psyc,ho/ogy, 1975, 67, 170- 173. TULVING, E. Episodic and semantic memory. In E. Tulving & W. Donaldson (Eds.), Organization of memory. New York: Academic Press, 1972. WINER, B. J. Stutisric~alprinciples in cxpc,rimerltu/ design. New York: McGraw-Hill, 1962. WITTROCK,M. C. The cognitive movement in instruction. Educutiona/ Psycho/ogist, 1978, 13, 15-29.
REFERENCE
NOTES
D. Thr role of informativc~jti~dbuck in productive, Icurrzing. Paper presented at the annual meeting of the American Educational Research Association, New York, 1977. 2. PHYE, G. D., & ENGEL, J. B. The udult Irurner: Learning to learn in an ucudcmic sptting. Manuscript submitted for publication, 1979. 1. PHYE, G.