- Email: [email protected]
Teaching students to summarize: Applying textlinguistics
System, Vol. 17, No. 1, pp. 1-11, 1989
0346-251X/89 $3.00 + 0.00 © 1989 Pergamon Press plc
Printed in Great Britain
TEACHING STUDENTS TO SUMMARIZE: APPLYING TEXTLINGUISTICS CAROL SHERRARD
School of Studies in Psychology, University of Bradford, United Kingdom The findings from research on written and oral summarization are discussed under three headings: summarizers' strategies, improving summarizers' performance, and the assessment of summary quality. Most research on summarizers' strategies has been based on the Macrostructure model of van Dijk and Kintsch, and provides broad confirmation of it. Clear-cut rules for summarization have been devised and successfully taught, although the central strategy, that of selecting important information, is least understood and most resistant to instruction. Summarizers' performance divides into two classes: " m a t u r e " performance involves a deep-level transformation of a text, departing from surface wording and the original order of propositions; " i m m a t u r e " performance is confined to the deletion, or verbatim retention, of surface elements. The objective measures of summary quality used by researchers are described, and ways in which these could be adapted for use in higher education assessment are indicated.
Textlinguistics is currently being fruitfully applied to the question of how people produce summaries. The research covers written and oral summaries of texts and discourse. Knowledge in this area should be useful in teaching summarizing skill to students of translation and interpreting. The purpose of this paper is to review the current research, and its relevance to three questions of interest to modern language teachers: what are the psychological strategies used by summarizers? How can summarizers' performance be improved? And how can summary quality be more validly and reliably assessed?
1. SUMMARIZERS' STRATEGIES The Macrostructure theory of Kintsch and van Dijk (1975, 1978; van Dijk, 1980) specifies four rules ("Macrorules" of Deletion, Selection, Generalization, and Construction) which are applied to the mental representation of a text in order to summarize it. It is a central claim of the theory that all the mental processes applied to a text, i.e. comprehension, recall, and summarization, are carried out on a single metal representation of the text. This mental representation is the Macrostructure, and the summary is said to be a direct expression of it.
2
C A R O L SHERRARD
Although this is not the only theory of summarization available (see, for instance, Rumelhart, 1977; Crothers, 1979; Schank and Abelson, 1977, Schank et al., 1980; Lehnert, 1981) it is the one that has given rise to most empirical research, possibly because it hypothesises such clear-cut rules and strategies in the summarizer. The Macrostructure is said to be built up during comprehension by first abstracting the explicit propositions from the text, then inferring propositions which are necessary to perceive cohesion in the text. Some propositions are next selected for storage in a limitedcapacity memory buffer over a series of cycles. At each cycle, only those propositions are retained in the buffer which can be related, through shared reference, with other propositions. In this way, propositions become linked together, and in turn become linked to superordinate propositions ("Macropropositions") which can be used to replace them if the text is to be stored in long-term memory or to be summarized. The Macropropositions are produced by applying the four Macrorules. The Deletion rule directs the deletion of inessential propositions. For example, " M a r y played with a ball. The ball was blue." becomes "'Mary played with a ball". The Selection rule directs the deletion of propositions which could be inferred from the remaining propositions, given knowledge of normal situations and conditions. Thus, " I went to Paris. So, I went to the station, bought a ticket, took the t r a i n . . . " becomes "'I went to Paris". The Generalization rule directs the substitution of a general term for a list of specific items: " M a r y played with a doll. Mary played with blocks." becomes "'Mary played with toys". Finally, the Construction rule directs the construction of a new proposition which allows deleted propositions to be inferred, again through knowledge of normal situations and conditions. Thus, " I went to the station, bought a t i c k e t . . . " are deleted and replaced with " / w e n t by train". The difference between the Construction rule and the Selection rule is that the Selection rule simply selects a proposition which already appears in the text, while the Construction rule, as its name indicates, requires the construction of a new proposition. The Selection and Deletion rules can operate on the text verbatim, while the Construction and Generalization rules require the generation of new forms. It may be noted, too, that two of the rules, Deletion and Generalization, lead to irrecoverable information loss; while Selection and Construction allow the reconstruction of deleted information through inferences about normality. The theory has been criticised on a number of grounds. Sanford and Garrod (1981) have pointed out that, although the theory has the appearance of a process model, it does not give a blow-by-blow account of how the Macrostructure is constructed by the reader. They also, like Schnotz (1983), criticise the subjective manner in which "propositions" are defined and identified. Finally, Kintsch himself (1982) has criticized his own and others' work for its use of narrative, rather then expository texts. The naturally occurring summarization task is mostly carried out on expository texts, but narrative texts have been used more
T E A C H I N G STUDENTS TO SUMMARIZE
3
often in psycholinguistic research because, having more transparent structure, they are more easily analysed. (For the same reason, they are more easily processed by readers, and especially, listeners.) In spite of these shortcomings, it has been possible for researchers to take the Macrostructure model and convert it into relatively straightforward empirical rules of summarization, corresponding to the four theoretical Macrorules. Brown and Day (1983) converted the Macrorules into the following four which, they hypothesized, would emerge developmentally in their order of difficulty in children's attempts to write summaries: DELETE trivial and redundant information; SELECT a topic sentence already in the text; SUBSTITUTE a general term for a list of objects or a sequence of actions; and INVENT a topic sentence, if one does not already appear in the text. Brown and Day (1983) did find that the youngest children in their sample (mean age 10 years, 7 months) all used the DELETE rule in their summaries of narrative texts, and that the other rules were only acquired later, and in their order of difficulty. However, the most difficult rule, INVENT a topic sentence, was not universally acquired. Even college students only used it on half the occasions when it would have been appropriate. "Experts" (graduate assistant teachers of Rhetoric), on the other hand, always used it. Garner (1982) also studied the performance of expert summarizers. While, like Brown and Day (1983) she found that only experts always used the INVENT rule, she additionally found that they were not conscious of doing so. (Brown and Day had found that, while experts did not spontaneously mention any of the summarization rules during an openended interview prior to the summarization task, when they were asked to verbalize while actually carrying the task out, they did report using procedures corresponding to the Macrorules, and in particular they focussed on the selection and construction of topic sentences.) Garner's study had hypothesized that summarization performance would be positively correlated with ability to verbalize the four Macrorules. This hypothesis was confirmed, with the exception that there was no relationship between the experts' use of the most difficult (INVENT) rule and their consciousness of using it. In another study, Brown et al. (1983) observed what they called a "mature strategy". This strategy emerged when mature summarizers (from age 16, and college students) were exceptionally pressed for space, and it consisted of ignoring the supplied topic sentence; instead "they combined across paragraphs and expressed the essential gist of large bodies of text in a few words" (p. 13). The severity of the space constraint imposed in this study, which was one tenth of the original narrative, will be appreciated when compared to the standard summary length of one third of the original text. The empirical work on written summarization generally supports the Macrostructure theory then, with the additional findings that use of the most difficult Macrorule (construction, or INVENT as some researchers term it) is neither universal nor conscious, and that mature performance may include, in addition to the use of all four Macrorules, the structural Mature Strategy described by Brown et al. (1983). Two studies have investigated summarization of speech (Kintsch and Kozminsky, 1977;
4
CAROL S H E R R A R D
Long and Harding-Esch, 1978). The latter is concerned specifically with summarizing from a second language into a first. Kintsch and Kozminsky (1977) found that summaries of speech were as good as summaries of text, even though repeated access to the input is not possible in the case of speech, and these summaries were therefore produced in less time. Although errors were rare in either type of summary, summaries of speech were more likely to have errors of addition in them. This study was confined to narratives, which are more easily processed than expository text or discourse because of their familiar structure. The result indicates, therefore, that, so long as the psychological and linguistic components of a task are easy, cognitive processing limitations are not apparent in summarizing speech. Long and Harding-Esch (1978) were particularly interested in decoding limitations on summarization, since they were comparing first- and second-language performance on this task in subjects with a high level of proficiency in their second language. They found the expected "second-language deficit" in the following aspects of the (first-language) written summaries of (second-language) expository speeches: fewer propositions were reproduced overall, but in particular fewer main propositions; more false propositions intruded. In general, summaries of second-language discourse were poor in their selectivity; they included less important information and more unimportant and inaccurate information. Long and Harding-Esch (1978) offer three hypotheses to account for the pattern of their results: a cognitive, a linguistic, and a production hypothesis. Firstly, they suggest that the reduced transmission of main propositions is accounted for by a lesser COGNITIVE competence to select and organise information when working in a second language. Secondly, that the greater number of false propositions is due to a specifically LINGUISTIC comprehension deficit in the second language. Thirdly, that the overall reduction in propositions is due to a PRODUCTION deficit in the second language. The absence of any studies of oral summarization of speech, which would be of direct use to teachers of interpreting, is a gap in the research literature.
2. IMPROVING SUMMARIZERS' PERFORMANCE Hare and Borchardt (1984) attempted to teach summarization skill directly to American high school students, using Brown and Day's (1983) empirical versions of the Macrorules, and additional rules to encompass Brown et al. "s (1983) Mature Strategy (i.e. a "Collapse Paragraphs" rule) and the need, indentified by Winograd (1982), to "polish the summary" (i.e. a rule stating "Create a natural-sounding summary. Adjustments may include paraphrasing, the insertion of connecting words like 'and' or 'because', and the insertion of introductory or closing s t a t e m e n t s . . . " ; Hare and Borchardt, 1984: p. 66). After three instructional sessions they found some improvement in both summary product and summary rule-usage. The summaries produced by the experimental group included more judged-important ideas, and were more efficient (as measured by the number of words used) than those produced by a control group. Usage of the rules tended to improve, either from non-usage to some usage, or from inconsistent usage to consistent usage. However,
T E A C H I N G STUDENTS TO SUMMARIZE
5
usage of the Construction Macrorule (the most difficult Macrorule, which requires the generation of a new proposition) did not improve significantly, even in cases where the subject was using the Mature Strategy. Hare and Borchardt (1984) conclude from their study that sensitivity to the important information in a text is a central feature of summarization skill, but one which is'the least understood and the most resistant to direct instruction. Other studies which have attempted to improve summarization performance have done so by making the signals of text structure more salient to the summarizer. Kieras (1985) has pointed out that readers lack a single schema for the structure of expository text (whereas they have a well-established schema for narratives). Readers of expository text should therefore be more dependent on surface structure cues. Britton et al. (1982) have reasoned that, given the consequently heavier cognitive demaads of expository text processing, any "prefetching" (priming of information relevant to text interpretation) should lighten the cognitive load of expository text reading. Text-structure signalling should facilitate prefetching. Given the plausibility of this argument, it is surprising to find that studies of the effect of signalling on expository text comprehension and recall have had generally negative findings, except when readers were either very young or unskilled (Roen and Piche, 1984). The only positive finding has been that signalling can increase the amount of spare cognitive processing capacity available, as measured by its effect in enhancing performance of a concurrent task (Britton et al., 1982) or in reducing reading time (Roen and Piche, 1984). Garner and McCaleb (1985) have, however, found that summarization performance can be improved by signalling in the text to be summarized. In this study, semantic and lexical signalling were used. (Semantic signalling consists of, for example, placing explicit topic sentences within paragraphs and providing an explicit thesis statement for the entire text. Lexical signalling consists of using words such as "important" with the information to be signalled). Garner and McCaleb (1985) found that such signalling had strong effects on summarizers' performance: the number of judged-important ideas they included was increased, and their integration of textual information (i.e. integration of several judgedimportant ideas within a single summary sentence) was enhanced. On the basis of their findings, Garner and McCaleb (1985) argue that summarization performance could be improved by directing students' attention to the textual signals which mark important information. These findings, considered in conjunction with the previous ones that signalling has the effect of increasing cognitive processing capacity (rather than quality) suggest that the major performance problem in summarization may be a limitation on cognitive capacity (i.e. a quantitative limitation), rather than a direct limitation on the quality of performance. On the other hand, Loman and Mayer (1983) have found that problem-solving performance can be improved after reading a signalled version of a text. They interpret their result to mean that signalling enhances the reader's understanding of the text's conceptual structure (i.e. inferences such as cause-effect links within the text). However; the improved performance they found could also be explained as due to the release of spare capacity by signalling. Whichever explanation is correct, there is sufficient evidence of signalling effects to justify teaching text signalling as an aid to summarization.
6
CAROL SHERRARD
3. ASSESSING S U M M A R Y Q U A L I T Y There is considerable uncertainty about what constitutes a good summary. The use of agreement between judges as an attempt at an objective measure of quality has not been successful. Brown and Day (1983) found that judges asked to make a global assessment of summary quality were unduly influenced by writing style. Sherrard (1985) found that agreement between judges was p o o r for such global assessments, and in cases where there was agreement it was in conflict with other empirical measures. Most researchers have used some measure o f important-information inclusion, which is an obvious criterion, but most have also felt the need for some structural criterion as well. However, we shall see that it is not obvious what the structural features of a good summary are, and they are consequently even less clearly defined. Since so m a n y researchers have used Kintsch and van Dijk's Macrostructure model with its postulated Macrorules, the extent to which a summary observes these rules has also been used as a criterion of quality, although this procedure is somewhat circular, since the studies have been attempting, at the same time, to verify that the Macrorules are a feature of real psychological processing. What is needed is an independent criterion of quality against which the Macrorules can be tested as a theory of summarization. Only Winograd (1984) looks at the interrelation between several measures of s u m m a r y performance.
3.1. Criteria o f summary quality: content criteria Content criteria of summary quality have generally used some consensus judgment measure of the important information in a text, and of whether it has been reproduced in the summary, using either independent judges or agreement among the summarizers themselves. Using the latter type of measure, Kintsch and Kozminsky (1977) defined " a good summary [as] one in which everyone agrees; when everyone summarizes a text differently, this indicates that they really d o n ' t know what to d o " . Garner (1982) used a similar device, based on Johnson (1970). In her study, each sentence of the text to be summarized was rated by 17 independent judges for its importance to a potential summary on a 3-point scale (3 = important, should appear in a summary; 2 -- moderately important, might or might not appear; 1 = unimportant and should not appear). The identification of " i d e a s " , and judgement as to their reproduction in the summaries, was carried out by Garner herself and one other judge, whose agreement turned out to be unanimous. Clearly, either of these procedures would be impractical for language teachers faced with the task of assessing students' summaries. More practical, possibly, would be the related method used by Sj6strom and H a r e (1984). They used only three raters for a procedure based on Garner's, though rating independent clauses, rather than whole sentences, for their importance to a potential summary. Whether the judged-important text clauses were reproduced in the summaries was assessed by two judges, whose correlation o f agreement was very high at 0.95 (compare Garner's unanimous result for the corresponding assessment in her study). The use of two judges in both Garner's and Sj6strom and H a r e ' s studies to assess the reproduction of important information in summaries is no more arduous, but is more systematic, than the double-marking system currently used for examination
T E A C H I N G STUDENTS TO SUMMARIZE
7
assessment in British higher education. While Garner used 17 judges to initially identify important text information, Sj6strom and Hare achieved comparable results with only three judges. Such a procedure should not be too arduous at least for examination purposes, and a reliable result could probably be achieved between two examination markers. Winograd (1984) used an amalgam of inter-summarizer agreement and an external criterion of quality as his measure. In a developmental study, the summaries produced by children were scored by measuring the extent to which they were similar to summaries o f the same texts produced by adults. A point-biserial correlation was obtained between the child's inclusion (1) or exclusion (0) of each text sentence, and the proportion of adults doing likewise. This method assumed, without any independent verification, that these adults' performance was superior to the children's. However, it suggests an alternative method for use by examiners: a model summary could be produced, and the point-biserial correlation between a student's summary and this model corpus could then be obtained as a measure of summary quality. 3.2 Criteria o f summary quality: structure The structural features of a good summary are much more difficult to specify, even if it is possible to point to certain global qualities that it may possess. One such quality, difficult to specify concretely, reflects a relatively holistic, "deep level" attack on the text by the summarizer. For instance, Hare and Borchardt (1984) comment that: "Younger a n d / o r poorer readers are more inclined to make decisions about summary inclusions and deletions on a piecemeal, sentence-by-sentence basis, whereas older a n d / o r good readers make their judgments based upon the meaning of the whole text" (p.63), yet their own quality criteria do not capture this distinction. Similarly, Brown and Day's (1983) Mature Strategy identifies a property of a good summary which their study had not anticipated, and which they therefore did not define more precisely than "combining across paragraphs and expressing the essential gist of large bodies of text in few words". (We shall see below that later researchers have tried to operationalize the Mature Strategy). Language teachers will recognise the associated problems of students who do not read the whole text before attempting to translate, or even understand, individual sentences, and those, probably the same students, who adopt sentence-by-sentence or even word-by-word translation strategies. Kintsch and van Dijk's notion of the Macrostructure may be an aid in teaching more holistic strategies, because it makes explicit and concrete the otherwise rather vague notion o f "the meaning of the whole text". Because of this difficulty in precise definition of structural summary quality, researchers have not agreed in the structural criteria they use, yet certain similarities emerge when their measures are compared. Brown et at.'s (1983) Mature Strategy and Kintsch and van Dijk's (1978) Construction rule are echoed in Winograd's " C o m b i n a t i o n s " and " I n v e n t i o n s " , respectively. "Combinations" are instances o f the combination of multiple text sentences into a single summary sentence. "Inventions" are single summary sentences which "conveyed the meaning of a paragraph, several paragraphs, or even the whole p a s s a g e . . . it was very difficult to tie them to any specific elements in the surface structure of the original sentences" (Winograd, 1984: p. 408). Garner and McCaleb's (1985) "Integration" criterion is similar to Winograd's Combinations,
8
CAROL SHERRARD
and therefore also akin to Brown and Day's Mature Strategy. The difference is that Integration combines a structural measure with a content measure, whereas Combinations and the Mature Strategy are, as defined, purely structural measures. For their Integration measure, Garner and McCaleb (1985) indentified five levels at which important text ideas were combined into single summary sentences: 0 (no important ideas reproduced), 1 (important ideas in separate sentences), 2 (two important ideas combined in a single sentence), 3 (three important ideas combined in a single sentence), and 4 (all important ideas combined in a single sentence). Garner and McCaleb (1985) noted, to their surprise, that reduction constraints did not increase Integration level. That is, having to compress the summary into fewer words did not result in combining more important text ideas within single summary sentences. Sherrard (1986) similarly found that there was no correlation between what she termed text-sentence "Domain", i.e. the number of text sentences embraced by a summary sentence, and the length of the text. The independence of Integration from the relatively surface feature of text length suggests that it is a genuine reflection of cognitive strategy, and therefore of quality. This suggestion is supported in a developmental study by Garner (1985), who noted that even undergraduates do not use many opportunities to integrate (this being partly responsible for their lesser succinctness than younger students, in spite of their greater ability to select important information). Garner and McCaleb (1985) also used as a structural criterion the "deviation in text-tosummary order of presentation of important ideas". This is the same as one component of the Mature Strategy, as described (slightly differently than in the previous quotation from them) in Brown and Day (1983): "combining across paragraphs, re-arranging by topic duster, and stating the gist in their own words" (p. 2). Here, "re-arranging by topic cluster" is added to the previously quoted description. The association between good performance and this particular structural measure has also been noted by other researchers. Basham and Rounds (1984) noted that some non-Western foreign students have a particular difficulty in writing summaries of academic articles which is manifested in "following the sequential development of the primary text", and in failing to appreciate that "it is possible to separate words from meaning--that is, that one can paraphrase someone else's words" (p. 527). Here Basham and Rounds link together the ability to depart from the text sequence with the ability to process the text at the level of gist, rather than surface wording, in much the same way that Brown's Mature Strategy does. Making this link makes it clear that it is not re-sequencing alone which indicates a good summary, for incompetence might also lead to re-sequencing. The re-sequencing must be appropriate. It is clear, indeed, that none of the structural criteria alone can stand as a measure of quality; there must always be, as well, some measure of the summarizer's success in transmitting the text gist. Two studies Winograd (1984) and Sherrard (1985), have attempted to systematically relate structural criteria to other criteria of quality. Winograd (1984) found a significant correlation between the frequency of Combinations, and the extent of a child's summary's agreement in content with a corpus of adult summaries. This correlation suggests that Winograd's Combinations, Garner's Integration, and Sherrard's Domain--all similar measures--may access a genuine structural feature of summary quality. Further evidence supporting this suggestion was reported in Sherrard (1985) who found that, for a sample of summaries written by British undergraduates, Domain was negatively correlated with text difficulty, and positively correlated with paragraph-combining.
T E A C H I N G STUDENTS TO SUMMARIZE
9
4. CONCLUSION A good deal is known, then, about the likely psycholinguistic strategies occurring in summarization. A clear distinction appears between what might be termed "mature" and "immature" performance. The mature summarizer is able to generate new propositions which condense together stretches of text as large as paragraphs, and in doing so departs freely from the original wording and sequence of propositions. This strategy is essentially holistic and "deep-structure". In contrast, the immature summarizer adopts a "copydelete" strategy, simply deleting or retaining propositions verbatim in their original order of occurrence. The immature strategy is essentially piecemeal and "surface-structure". The strategies of mature performance are normally outside the summarizer's awareness, therefore one teaching policy might be to make them explicit for the student. The Macrorules of Kintsch and van Dijk, as converted into empirical equivalents by Brown and her colleagues, and as supplemented by the Collapse Paragraphs and Polishing rules in Hare and Borchardt's work, may be an aid in such teaching. Summarization of speech, which does not allow repeated access to the input, does not in itself result in performance limitations, as Kintsch and Kozminsky (1977) have shown. Performance is in fact more rapid because repeated scanning of the input is not possible. However, the extent to which the ease of the text was responsible for Kintsch and Kozminsky's results is not known, since they used narratives only. Cross-language summarization has certain special features. Working in a non-native language places an extra load on cognitive processing and, especially when the text is difficult, this may be manifested in poorer comprehension, reduced overall information transmission, and consquently reduced ability to select important information. Selectivity is also identified as a major difficulty in native-language performance. Hare and Borchardt (1984) found that both selectivity and another major difficulty, the construction of new propositions, were resistant to direct instruction. However, Garner and McCaleb (1985) were able to improve summarization performance by signalling the important information in a text. Evidently, it may be insufficient merely to instruct students in how to condense important text information: they may need instruction in how to identify the important information in the first place. Although it is not feasible for teachers to add signalling to all their instructional texts, it should be possible, as Garner and McCaleb (1985) suggest, to train students how to identify important information by alerting them to the function and form of textual signals. It seems likely that the signalling of important information has its effect on summarization performance by releasing cognitive resources otherwise used for text comprehension. Clearly, given the mature/immature distinction in summarization performance which emerges from the literature, measures of summary quality would be expected to relate to this. Mature performance involves both content and structure. More than this, content and structure can be optimally integrated in an economic structuring of appropriate content. However, for assessment purposes, content and structure can separated, and credit awarded for good performance in one aspect in the absence of the other. For example, Hare and
10
CAROL SHERRARD
Borchardt (1984) found that part of Brown and Day's "Mature Strategy" (Collapsing text paragraphs together) could be attained before appropriate selectivity. On the other hand, Garner (1985) found that older students may surpass young ones in selectivity, while failing to compress the selected propositions to the fullest extent possible (which resulted in their conforming less well than younger students to length limits). Some of the content measures which have been used by researchers lend themselves readily to assessment purposes. All the measures are consensual, but the irreducible element of subjectivity in this can be made into a more reliable judgement by obtaining a prior consensus between markers regarding the important ideas in a given text before assessment of students' summaries is carried out. Once the propositions in a text have been identified, it is also possible to apply one or other of the purely structural criteria of quality, such as paragraph-collapsing, or Winograd's Combinations, if content and structure of summaries are to be assessed separately. Finally, Garner and McCaleb's "Integration Level" measure offers a convenient way of obtaining, in a single measure, an assessment of both selectivity and compression.
REFERENCES BASHAM, C. and ROI3NDS, P. (1984) A discourse analysis approach to summary writing. Tesol Quarterly 18, 527. BRITTON, B. K., GLYNN, S. M., MEYER, B. J. F. and PENLAND, M. J. (1982) Effects of text structure on use of cognitive capacity during reading. Journal of Educational Psychology 74, 51-61. BROWN, A. and DAY, J. D. (1983) Macrorules for summarizing texts: the development of expertise. Journal of Verbal Learning and Verbal Behaviour 22, 1-4. BROWN, A. L., DAY, J. D. and JONES, R. S. (1983) Development of plans for summarizing texts: the development of expertise. Child Development 54, 968-679. CROTHERS, E. J. (1979) Paragraph Structure Inference. Norwood, NJ U.S.A.: Ablex. van DIJK, T. A. (1980) Macrostructures. Hillsdale, NJ U.S.A.: Erlbaum. GARNER, R. (1982) Efficient text summarization: costs and benefits. Journal of Educational Research 75, 275-279. GARNER, R. (1985) Text summarization deficiencies among older students: awareness or production ability? American Educational Research Journal 22, 549-560. GARNER, R. and McCALEB, J. L. (1985) Effects of text manipulations on quality of written summaries. Contemporary Educational Psychology 10, 139-149. HARE, V. C. and BORCHARDT, K. M. (1984) Direct instruction of summarization skills. Reading Research Quarterly Fall, XX/1, 62-78. JOHNSON, R. E. (1970) Recall of prose as a function of the structural importance of the linguistic units. Journal of Verbal Learning and Verbal Behaviour 9, 12-20. KIERAS, D. E. (1985) Thematic processes in the comprehension of technical prose. In Britton, B. K. and Black J. B. (eds), Understanding Expository Text, pp. 89-107. Hillsdale, NJ U.S.A.: Erlbaum. KINTSCH, W. (1982) Text representations. In Otto, W. and White, S. (eds), Reading Expository Material, pp. 87-101. New York: Academic Press. KINTSCH, W. and van DIJK, T. A. (1975) Comment on se rappelle et on resume des histoires. Languages 40, 98-116. KINTSCH, W. and van DIJK, T. A. (1978) Toward a model of discourse comprehension and production. Psychological Review 85, 363-394. KINTSCH, W. and KOZMINSKY, E. (1977) Summarizing stories after reading and listening. Journal of Educational Psychology 5, 491-499. LEHNERT, W. G. (1981) Plot units and narrative summarization. Cognitive Science October-November, 293-331. LOMAN, N. L. and MAYER, R. E. (1983) Signalling techniques that increase the understandability of expository prose. Journal of Educational Psychology 75, 401-412. LONG, J. and HARDING-ESCH, E. (1978) Summary and recall of text in first and second languages: some factors contributing to performance differences. In Gerver, D. and Sinaiko, W. (eds), Language Interpretation and Communication, pp. 273-287. New York: Plenum.
TEACHING STUDENTS TO SUMMARIZE
ll
ROEN, D. H. and PICHE, G. L. (1984) The effects of selected text-forming structures on college freshmen's comprehension of expository prose. Research In The Teaching of English 18, 8-25. RUMELHART, D. E. (1977) Understanding and summarizing brief stories. In Laberge, D. and Samuels, S. (eds), Basic Processes in Reading, Perception, and Comprehension. Hillsdale, NJ U.S.A.: Erlbaum. SANFORD, A. J. and GARROD, S. C. (1981) Understanding Written Language. New York: Wiley. SCHANK, R. and ABELSON, R. (1977) Scripts, Plans, Goals and Understanding." An Enquiry Into Human Knowledge Structures. Hillsdale, NJ U.S.A.: Erlbaum. SCHANK. R., LEBOWITZ, M. and BIRNBAUM, L. (1980) An integrated understander. American Journal of Computational Linguistics 6, 13-30. SCHNOTZ, W. (1983) On the influence of text organization on learning outcomes. In Rickheit, G. and Bock, M. (eds), Psycholinguistic Studies in Language Processing. Berlin: de Gruyter. SHERRARD, C. A. (1985) Collapsing paragraphs: the engineer's writing problem? Paper presented at Meeting of the Cognitive Psychology Section, British Psychological Society, Oxford, September 1985. SHERRARD, C. A. (1986) Summary-writing: a topographical study. Written Communication 3, 25-30. SJOSTROM, C. L. and HARE, V.C. (1984) Teaching high school students to identify main ideas in expository text. Journal of Educational Research 78, 114-118. WINOGRAD, P. N. (1982) An examination of strategic differences in summarizing texts. Unpublished doctoral dissertation, University of Illinois, Urbana. WINOGRAD, P. N. (1984) Strategic difficulties in summarizing texts. Reading Research Quarterly 19, 404-425.
0346-251X/89 $3.00 + 0.00 © 1989 Pergamon Press plc
Printed in Great Britain
TEACHING STUDENTS TO SUMMARIZE: APPLYING TEXTLINGUISTICS CAROL SHERRARD
School of Studies in Psychology, University of Bradford, United Kingdom The findings from research on written and oral summarization are discussed under three headings: summarizers' strategies, improving summarizers' performance, and the assessment of summary quality. Most research on summarizers' strategies has been based on the Macrostructure model of van Dijk and Kintsch, and provides broad confirmation of it. Clear-cut rules for summarization have been devised and successfully taught, although the central strategy, that of selecting important information, is least understood and most resistant to instruction. Summarizers' performance divides into two classes: " m a t u r e " performance involves a deep-level transformation of a text, departing from surface wording and the original order of propositions; " i m m a t u r e " performance is confined to the deletion, or verbatim retention, of surface elements. The objective measures of summary quality used by researchers are described, and ways in which these could be adapted for use in higher education assessment are indicated.
Textlinguistics is currently being fruitfully applied to the question of how people produce summaries. The research covers written and oral summaries of texts and discourse. Knowledge in this area should be useful in teaching summarizing skill to students of translation and interpreting. The purpose of this paper is to review the current research, and its relevance to three questions of interest to modern language teachers: what are the psychological strategies used by summarizers? How can summarizers' performance be improved? And how can summary quality be more validly and reliably assessed?
1. SUMMARIZERS' STRATEGIES The Macrostructure theory of Kintsch and van Dijk (1975, 1978; van Dijk, 1980) specifies four rules ("Macrorules" of Deletion, Selection, Generalization, and Construction) which are applied to the mental representation of a text in order to summarize it. It is a central claim of the theory that all the mental processes applied to a text, i.e. comprehension, recall, and summarization, are carried out on a single metal representation of the text. This mental representation is the Macrostructure, and the summary is said to be a direct expression of it.
2
C A R O L SHERRARD
Although this is not the only theory of summarization available (see, for instance, Rumelhart, 1977; Crothers, 1979; Schank and Abelson, 1977, Schank et al., 1980; Lehnert, 1981) it is the one that has given rise to most empirical research, possibly because it hypothesises such clear-cut rules and strategies in the summarizer. The Macrostructure is said to be built up during comprehension by first abstracting the explicit propositions from the text, then inferring propositions which are necessary to perceive cohesion in the text. Some propositions are next selected for storage in a limitedcapacity memory buffer over a series of cycles. At each cycle, only those propositions are retained in the buffer which can be related, through shared reference, with other propositions. In this way, propositions become linked together, and in turn become linked to superordinate propositions ("Macropropositions") which can be used to replace them if the text is to be stored in long-term memory or to be summarized. The Macropropositions are produced by applying the four Macrorules. The Deletion rule directs the deletion of inessential propositions. For example, " M a r y played with a ball. The ball was blue." becomes "'Mary played with a ball". The Selection rule directs the deletion of propositions which could be inferred from the remaining propositions, given knowledge of normal situations and conditions. Thus, " I went to Paris. So, I went to the station, bought a ticket, took the t r a i n . . . " becomes "'I went to Paris". The Generalization rule directs the substitution of a general term for a list of specific items: " M a r y played with a doll. Mary played with blocks." becomes "'Mary played with toys". Finally, the Construction rule directs the construction of a new proposition which allows deleted propositions to be inferred, again through knowledge of normal situations and conditions. Thus, " I went to the station, bought a t i c k e t . . . " are deleted and replaced with " / w e n t by train". The difference between the Construction rule and the Selection rule is that the Selection rule simply selects a proposition which already appears in the text, while the Construction rule, as its name indicates, requires the construction of a new proposition. The Selection and Deletion rules can operate on the text verbatim, while the Construction and Generalization rules require the generation of new forms. It may be noted, too, that two of the rules, Deletion and Generalization, lead to irrecoverable information loss; while Selection and Construction allow the reconstruction of deleted information through inferences about normality. The theory has been criticised on a number of grounds. Sanford and Garrod (1981) have pointed out that, although the theory has the appearance of a process model, it does not give a blow-by-blow account of how the Macrostructure is constructed by the reader. They also, like Schnotz (1983), criticise the subjective manner in which "propositions" are defined and identified. Finally, Kintsch himself (1982) has criticized his own and others' work for its use of narrative, rather then expository texts. The naturally occurring summarization task is mostly carried out on expository texts, but narrative texts have been used more
T E A C H I N G STUDENTS TO SUMMARIZE
3
often in psycholinguistic research because, having more transparent structure, they are more easily analysed. (For the same reason, they are more easily processed by readers, and especially, listeners.) In spite of these shortcomings, it has been possible for researchers to take the Macrostructure model and convert it into relatively straightforward empirical rules of summarization, corresponding to the four theoretical Macrorules. Brown and Day (1983) converted the Macrorules into the following four which, they hypothesized, would emerge developmentally in their order of difficulty in children's attempts to write summaries: DELETE trivial and redundant information; SELECT a topic sentence already in the text; SUBSTITUTE a general term for a list of objects or a sequence of actions; and INVENT a topic sentence, if one does not already appear in the text. Brown and Day (1983) did find that the youngest children in their sample (mean age 10 years, 7 months) all used the DELETE rule in their summaries of narrative texts, and that the other rules were only acquired later, and in their order of difficulty. However, the most difficult rule, INVENT a topic sentence, was not universally acquired. Even college students only used it on half the occasions when it would have been appropriate. "Experts" (graduate assistant teachers of Rhetoric), on the other hand, always used it. Garner (1982) also studied the performance of expert summarizers. While, like Brown and Day (1983) she found that only experts always used the INVENT rule, she additionally found that they were not conscious of doing so. (Brown and Day had found that, while experts did not spontaneously mention any of the summarization rules during an openended interview prior to the summarization task, when they were asked to verbalize while actually carrying the task out, they did report using procedures corresponding to the Macrorules, and in particular they focussed on the selection and construction of topic sentences.) Garner's study had hypothesized that summarization performance would be positively correlated with ability to verbalize the four Macrorules. This hypothesis was confirmed, with the exception that there was no relationship between the experts' use of the most difficult (INVENT) rule and their consciousness of using it. In another study, Brown et al. (1983) observed what they called a "mature strategy". This strategy emerged when mature summarizers (from age 16, and college students) were exceptionally pressed for space, and it consisted of ignoring the supplied topic sentence; instead "they combined across paragraphs and expressed the essential gist of large bodies of text in a few words" (p. 13). The severity of the space constraint imposed in this study, which was one tenth of the original narrative, will be appreciated when compared to the standard summary length of one third of the original text. The empirical work on written summarization generally supports the Macrostructure theory then, with the additional findings that use of the most difficult Macrorule (construction, or INVENT as some researchers term it) is neither universal nor conscious, and that mature performance may include, in addition to the use of all four Macrorules, the structural Mature Strategy described by Brown et al. (1983). Two studies have investigated summarization of speech (Kintsch and Kozminsky, 1977;
4
CAROL S H E R R A R D
Long and Harding-Esch, 1978). The latter is concerned specifically with summarizing from a second language into a first. Kintsch and Kozminsky (1977) found that summaries of speech were as good as summaries of text, even though repeated access to the input is not possible in the case of speech, and these summaries were therefore produced in less time. Although errors were rare in either type of summary, summaries of speech were more likely to have errors of addition in them. This study was confined to narratives, which are more easily processed than expository text or discourse because of their familiar structure. The result indicates, therefore, that, so long as the psychological and linguistic components of a task are easy, cognitive processing limitations are not apparent in summarizing speech. Long and Harding-Esch (1978) were particularly interested in decoding limitations on summarization, since they were comparing first- and second-language performance on this task in subjects with a high level of proficiency in their second language. They found the expected "second-language deficit" in the following aspects of the (first-language) written summaries of (second-language) expository speeches: fewer propositions were reproduced overall, but in particular fewer main propositions; more false propositions intruded. In general, summaries of second-language discourse were poor in their selectivity; they included less important information and more unimportant and inaccurate information. Long and Harding-Esch (1978) offer three hypotheses to account for the pattern of their results: a cognitive, a linguistic, and a production hypothesis. Firstly, they suggest that the reduced transmission of main propositions is accounted for by a lesser COGNITIVE competence to select and organise information when working in a second language. Secondly, that the greater number of false propositions is due to a specifically LINGUISTIC comprehension deficit in the second language. Thirdly, that the overall reduction in propositions is due to a PRODUCTION deficit in the second language. The absence of any studies of oral summarization of speech, which would be of direct use to teachers of interpreting, is a gap in the research literature.
2. IMPROVING SUMMARIZERS' PERFORMANCE Hare and Borchardt (1984) attempted to teach summarization skill directly to American high school students, using Brown and Day's (1983) empirical versions of the Macrorules, and additional rules to encompass Brown et al. "s (1983) Mature Strategy (i.e. a "Collapse Paragraphs" rule) and the need, indentified by Winograd (1982), to "polish the summary" (i.e. a rule stating "Create a natural-sounding summary. Adjustments may include paraphrasing, the insertion of connecting words like 'and' or 'because', and the insertion of introductory or closing s t a t e m e n t s . . . " ; Hare and Borchardt, 1984: p. 66). After three instructional sessions they found some improvement in both summary product and summary rule-usage. The summaries produced by the experimental group included more judged-important ideas, and were more efficient (as measured by the number of words used) than those produced by a control group. Usage of the rules tended to improve, either from non-usage to some usage, or from inconsistent usage to consistent usage. However,
T E A C H I N G STUDENTS TO SUMMARIZE
5
usage of the Construction Macrorule (the most difficult Macrorule, which requires the generation of a new proposition) did not improve significantly, even in cases where the subject was using the Mature Strategy. Hare and Borchardt (1984) conclude from their study that sensitivity to the important information in a text is a central feature of summarization skill, but one which is'the least understood and the most resistant to direct instruction. Other studies which have attempted to improve summarization performance have done so by making the signals of text structure more salient to the summarizer. Kieras (1985) has pointed out that readers lack a single schema for the structure of expository text (whereas they have a well-established schema for narratives). Readers of expository text should therefore be more dependent on surface structure cues. Britton et al. (1982) have reasoned that, given the consequently heavier cognitive demaads of expository text processing, any "prefetching" (priming of information relevant to text interpretation) should lighten the cognitive load of expository text reading. Text-structure signalling should facilitate prefetching. Given the plausibility of this argument, it is surprising to find that studies of the effect of signalling on expository text comprehension and recall have had generally negative findings, except when readers were either very young or unskilled (Roen and Piche, 1984). The only positive finding has been that signalling can increase the amount of spare cognitive processing capacity available, as measured by its effect in enhancing performance of a concurrent task (Britton et al., 1982) or in reducing reading time (Roen and Piche, 1984). Garner and McCaleb (1985) have, however, found that summarization performance can be improved by signalling in the text to be summarized. In this study, semantic and lexical signalling were used. (Semantic signalling consists of, for example, placing explicit topic sentences within paragraphs and providing an explicit thesis statement for the entire text. Lexical signalling consists of using words such as "important" with the information to be signalled). Garner and McCaleb (1985) found that such signalling had strong effects on summarizers' performance: the number of judged-important ideas they included was increased, and their integration of textual information (i.e. integration of several judgedimportant ideas within a single summary sentence) was enhanced. On the basis of their findings, Garner and McCaleb (1985) argue that summarization performance could be improved by directing students' attention to the textual signals which mark important information. These findings, considered in conjunction with the previous ones that signalling has the effect of increasing cognitive processing capacity (rather than quality) suggest that the major performance problem in summarization may be a limitation on cognitive capacity (i.e. a quantitative limitation), rather than a direct limitation on the quality of performance. On the other hand, Loman and Mayer (1983) have found that problem-solving performance can be improved after reading a signalled version of a text. They interpret their result to mean that signalling enhances the reader's understanding of the text's conceptual structure (i.e. inferences such as cause-effect links within the text). However; the improved performance they found could also be explained as due to the release of spare capacity by signalling. Whichever explanation is correct, there is sufficient evidence of signalling effects to justify teaching text signalling as an aid to summarization.
6
CAROL SHERRARD
3. ASSESSING S U M M A R Y Q U A L I T Y There is considerable uncertainty about what constitutes a good summary. The use of agreement between judges as an attempt at an objective measure of quality has not been successful. Brown and Day (1983) found that judges asked to make a global assessment of summary quality were unduly influenced by writing style. Sherrard (1985) found that agreement between judges was p o o r for such global assessments, and in cases where there was agreement it was in conflict with other empirical measures. Most researchers have used some measure o f important-information inclusion, which is an obvious criterion, but most have also felt the need for some structural criterion as well. However, we shall see that it is not obvious what the structural features of a good summary are, and they are consequently even less clearly defined. Since so m a n y researchers have used Kintsch and van Dijk's Macrostructure model with its postulated Macrorules, the extent to which a summary observes these rules has also been used as a criterion of quality, although this procedure is somewhat circular, since the studies have been attempting, at the same time, to verify that the Macrorules are a feature of real psychological processing. What is needed is an independent criterion of quality against which the Macrorules can be tested as a theory of summarization. Only Winograd (1984) looks at the interrelation between several measures of s u m m a r y performance.
3.1. Criteria o f summary quality: content criteria Content criteria of summary quality have generally used some consensus judgment measure of the important information in a text, and of whether it has been reproduced in the summary, using either independent judges or agreement among the summarizers themselves. Using the latter type of measure, Kintsch and Kozminsky (1977) defined " a good summary [as] one in which everyone agrees; when everyone summarizes a text differently, this indicates that they really d o n ' t know what to d o " . Garner (1982) used a similar device, based on Johnson (1970). In her study, each sentence of the text to be summarized was rated by 17 independent judges for its importance to a potential summary on a 3-point scale (3 = important, should appear in a summary; 2 -- moderately important, might or might not appear; 1 = unimportant and should not appear). The identification of " i d e a s " , and judgement as to their reproduction in the summaries, was carried out by Garner herself and one other judge, whose agreement turned out to be unanimous. Clearly, either of these procedures would be impractical for language teachers faced with the task of assessing students' summaries. More practical, possibly, would be the related method used by Sj6strom and H a r e (1984). They used only three raters for a procedure based on Garner's, though rating independent clauses, rather than whole sentences, for their importance to a potential summary. Whether the judged-important text clauses were reproduced in the summaries was assessed by two judges, whose correlation o f agreement was very high at 0.95 (compare Garner's unanimous result for the corresponding assessment in her study). The use of two judges in both Garner's and Sj6strom and H a r e ' s studies to assess the reproduction of important information in summaries is no more arduous, but is more systematic, than the double-marking system currently used for examination
T E A C H I N G STUDENTS TO SUMMARIZE
7
assessment in British higher education. While Garner used 17 judges to initially identify important text information, Sj6strom and Hare achieved comparable results with only three judges. Such a procedure should not be too arduous at least for examination purposes, and a reliable result could probably be achieved between two examination markers. Winograd (1984) used an amalgam of inter-summarizer agreement and an external criterion of quality as his measure. In a developmental study, the summaries produced by children were scored by measuring the extent to which they were similar to summaries o f the same texts produced by adults. A point-biserial correlation was obtained between the child's inclusion (1) or exclusion (0) of each text sentence, and the proportion of adults doing likewise. This method assumed, without any independent verification, that these adults' performance was superior to the children's. However, it suggests an alternative method for use by examiners: a model summary could be produced, and the point-biserial correlation between a student's summary and this model corpus could then be obtained as a measure of summary quality. 3.2 Criteria o f summary quality: structure The structural features of a good summary are much more difficult to specify, even if it is possible to point to certain global qualities that it may possess. One such quality, difficult to specify concretely, reflects a relatively holistic, "deep level" attack on the text by the summarizer. For instance, Hare and Borchardt (1984) comment that: "Younger a n d / o r poorer readers are more inclined to make decisions about summary inclusions and deletions on a piecemeal, sentence-by-sentence basis, whereas older a n d / o r good readers make their judgments based upon the meaning of the whole text" (p.63), yet their own quality criteria do not capture this distinction. Similarly, Brown and Day's (1983) Mature Strategy identifies a property of a good summary which their study had not anticipated, and which they therefore did not define more precisely than "combining across paragraphs and expressing the essential gist of large bodies of text in few words". (We shall see below that later researchers have tried to operationalize the Mature Strategy). Language teachers will recognise the associated problems of students who do not read the whole text before attempting to translate, or even understand, individual sentences, and those, probably the same students, who adopt sentence-by-sentence or even word-by-word translation strategies. Kintsch and van Dijk's notion of the Macrostructure may be an aid in teaching more holistic strategies, because it makes explicit and concrete the otherwise rather vague notion o f "the meaning of the whole text". Because of this difficulty in precise definition of structural summary quality, researchers have not agreed in the structural criteria they use, yet certain similarities emerge when their measures are compared. Brown et at.'s (1983) Mature Strategy and Kintsch and van Dijk's (1978) Construction rule are echoed in Winograd's " C o m b i n a t i o n s " and " I n v e n t i o n s " , respectively. "Combinations" are instances o f the combination of multiple text sentences into a single summary sentence. "Inventions" are single summary sentences which "conveyed the meaning of a paragraph, several paragraphs, or even the whole p a s s a g e . . . it was very difficult to tie them to any specific elements in the surface structure of the original sentences" (Winograd, 1984: p. 408). Garner and McCaleb's (1985) "Integration" criterion is similar to Winograd's Combinations,
8
CAROL SHERRARD
and therefore also akin to Brown and Day's Mature Strategy. The difference is that Integration combines a structural measure with a content measure, whereas Combinations and the Mature Strategy are, as defined, purely structural measures. For their Integration measure, Garner and McCaleb (1985) indentified five levels at which important text ideas were combined into single summary sentences: 0 (no important ideas reproduced), 1 (important ideas in separate sentences), 2 (two important ideas combined in a single sentence), 3 (three important ideas combined in a single sentence), and 4 (all important ideas combined in a single sentence). Garner and McCaleb (1985) noted, to their surprise, that reduction constraints did not increase Integration level. That is, having to compress the summary into fewer words did not result in combining more important text ideas within single summary sentences. Sherrard (1986) similarly found that there was no correlation between what she termed text-sentence "Domain", i.e. the number of text sentences embraced by a summary sentence, and the length of the text. The independence of Integration from the relatively surface feature of text length suggests that it is a genuine reflection of cognitive strategy, and therefore of quality. This suggestion is supported in a developmental study by Garner (1985), who noted that even undergraduates do not use many opportunities to integrate (this being partly responsible for their lesser succinctness than younger students, in spite of their greater ability to select important information). Garner and McCaleb (1985) also used as a structural criterion the "deviation in text-tosummary order of presentation of important ideas". This is the same as one component of the Mature Strategy, as described (slightly differently than in the previous quotation from them) in Brown and Day (1983): "combining across paragraphs, re-arranging by topic duster, and stating the gist in their own words" (p. 2). Here, "re-arranging by topic cluster" is added to the previously quoted description. The association between good performance and this particular structural measure has also been noted by other researchers. Basham and Rounds (1984) noted that some non-Western foreign students have a particular difficulty in writing summaries of academic articles which is manifested in "following the sequential development of the primary text", and in failing to appreciate that "it is possible to separate words from meaning--that is, that one can paraphrase someone else's words" (p. 527). Here Basham and Rounds link together the ability to depart from the text sequence with the ability to process the text at the level of gist, rather than surface wording, in much the same way that Brown's Mature Strategy does. Making this link makes it clear that it is not re-sequencing alone which indicates a good summary, for incompetence might also lead to re-sequencing. The re-sequencing must be appropriate. It is clear, indeed, that none of the structural criteria alone can stand as a measure of quality; there must always be, as well, some measure of the summarizer's success in transmitting the text gist. Two studies Winograd (1984) and Sherrard (1985), have attempted to systematically relate structural criteria to other criteria of quality. Winograd (1984) found a significant correlation between the frequency of Combinations, and the extent of a child's summary's agreement in content with a corpus of adult summaries. This correlation suggests that Winograd's Combinations, Garner's Integration, and Sherrard's Domain--all similar measures--may access a genuine structural feature of summary quality. Further evidence supporting this suggestion was reported in Sherrard (1985) who found that, for a sample of summaries written by British undergraduates, Domain was negatively correlated with text difficulty, and positively correlated with paragraph-combining.
T E A C H I N G STUDENTS TO SUMMARIZE
9
4. CONCLUSION A good deal is known, then, about the likely psycholinguistic strategies occurring in summarization. A clear distinction appears between what might be termed "mature" and "immature" performance. The mature summarizer is able to generate new propositions which condense together stretches of text as large as paragraphs, and in doing so departs freely from the original wording and sequence of propositions. This strategy is essentially holistic and "deep-structure". In contrast, the immature summarizer adopts a "copydelete" strategy, simply deleting or retaining propositions verbatim in their original order of occurrence. The immature strategy is essentially piecemeal and "surface-structure". The strategies of mature performance are normally outside the summarizer's awareness, therefore one teaching policy might be to make them explicit for the student. The Macrorules of Kintsch and van Dijk, as converted into empirical equivalents by Brown and her colleagues, and as supplemented by the Collapse Paragraphs and Polishing rules in Hare and Borchardt's work, may be an aid in such teaching. Summarization of speech, which does not allow repeated access to the input, does not in itself result in performance limitations, as Kintsch and Kozminsky (1977) have shown. Performance is in fact more rapid because repeated scanning of the input is not possible. However, the extent to which the ease of the text was responsible for Kintsch and Kozminsky's results is not known, since they used narratives only. Cross-language summarization has certain special features. Working in a non-native language places an extra load on cognitive processing and, especially when the text is difficult, this may be manifested in poorer comprehension, reduced overall information transmission, and consquently reduced ability to select important information. Selectivity is also identified as a major difficulty in native-language performance. Hare and Borchardt (1984) found that both selectivity and another major difficulty, the construction of new propositions, were resistant to direct instruction. However, Garner and McCaleb (1985) were able to improve summarization performance by signalling the important information in a text. Evidently, it may be insufficient merely to instruct students in how to condense important text information: they may need instruction in how to identify the important information in the first place. Although it is not feasible for teachers to add signalling to all their instructional texts, it should be possible, as Garner and McCaleb (1985) suggest, to train students how to identify important information by alerting them to the function and form of textual signals. It seems likely that the signalling of important information has its effect on summarization performance by releasing cognitive resources otherwise used for text comprehension. Clearly, given the mature/immature distinction in summarization performance which emerges from the literature, measures of summary quality would be expected to relate to this. Mature performance involves both content and structure. More than this, content and structure can be optimally integrated in an economic structuring of appropriate content. However, for assessment purposes, content and structure can separated, and credit awarded for good performance in one aspect in the absence of the other. For example, Hare and
10
CAROL SHERRARD
Borchardt (1984) found that part of Brown and Day's "Mature Strategy" (Collapsing text paragraphs together) could be attained before appropriate selectivity. On the other hand, Garner (1985) found that older students may surpass young ones in selectivity, while failing to compress the selected propositions to the fullest extent possible (which resulted in their conforming less well than younger students to length limits). Some of the content measures which have been used by researchers lend themselves readily to assessment purposes. All the measures are consensual, but the irreducible element of subjectivity in this can be made into a more reliable judgement by obtaining a prior consensus between markers regarding the important ideas in a given text before assessment of students' summaries is carried out. Once the propositions in a text have been identified, it is also possible to apply one or other of the purely structural criteria of quality, such as paragraph-collapsing, or Winograd's Combinations, if content and structure of summaries are to be assessed separately. Finally, Garner and McCaleb's "Integration Level" measure offers a convenient way of obtaining, in a single measure, an assessment of both selectivity and compression.
REFERENCES BASHAM, C. and ROI3NDS, P. (1984) A discourse analysis approach to summary writing. Tesol Quarterly 18, 527. BRITTON, B. K., GLYNN, S. M., MEYER, B. J. F. and PENLAND, M. J. (1982) Effects of text structure on use of cognitive capacity during reading. Journal of Educational Psychology 74, 51-61. BROWN, A. and DAY, J. D. (1983) Macrorules for summarizing texts: the development of expertise. Journal of Verbal Learning and Verbal Behaviour 22, 1-4. BROWN, A. L., DAY, J. D. and JONES, R. S. (1983) Development of plans for summarizing texts: the development of expertise. Child Development 54, 968-679. CROTHERS, E. J. (1979) Paragraph Structure Inference. Norwood, NJ U.S.A.: Ablex. van DIJK, T. A. (1980) Macrostructures. Hillsdale, NJ U.S.A.: Erlbaum. GARNER, R. (1982) Efficient text summarization: costs and benefits. Journal of Educational Research 75, 275-279. GARNER, R. (1985) Text summarization deficiencies among older students: awareness or production ability? American Educational Research Journal 22, 549-560. GARNER, R. and McCALEB, J. L. (1985) Effects of text manipulations on quality of written summaries. Contemporary Educational Psychology 10, 139-149. HARE, V. C. and BORCHARDT, K. M. (1984) Direct instruction of summarization skills. Reading Research Quarterly Fall, XX/1, 62-78. JOHNSON, R. E. (1970) Recall of prose as a function of the structural importance of the linguistic units. Journal of Verbal Learning and Verbal Behaviour 9, 12-20. KIERAS, D. E. (1985) Thematic processes in the comprehension of technical prose. In Britton, B. K. and Black J. B. (eds), Understanding Expository Text, pp. 89-107. Hillsdale, NJ U.S.A.: Erlbaum. KINTSCH, W. (1982) Text representations. In Otto, W. and White, S. (eds), Reading Expository Material, pp. 87-101. New York: Academic Press. KINTSCH, W. and van DIJK, T. A. (1975) Comment on se rappelle et on resume des histoires. Languages 40, 98-116. KINTSCH, W. and van DIJK, T. A. (1978) Toward a model of discourse comprehension and production. Psychological Review 85, 363-394. KINTSCH, W. and KOZMINSKY, E. (1977) Summarizing stories after reading and listening. Journal of Educational Psychology 5, 491-499. LEHNERT, W. G. (1981) Plot units and narrative summarization. Cognitive Science October-November, 293-331. LOMAN, N. L. and MAYER, R. E. (1983) Signalling techniques that increase the understandability of expository prose. Journal of Educational Psychology 75, 401-412. LONG, J. and HARDING-ESCH, E. (1978) Summary and recall of text in first and second languages: some factors contributing to performance differences. In Gerver, D. and Sinaiko, W. (eds), Language Interpretation and Communication, pp. 273-287. New York: Plenum.
TEACHING STUDENTS TO SUMMARIZE
ll
ROEN, D. H. and PICHE, G. L. (1984) The effects of selected text-forming structures on college freshmen's comprehension of expository prose. Research In The Teaching of English 18, 8-25. RUMELHART, D. E. (1977) Understanding and summarizing brief stories. In Laberge, D. and Samuels, S. (eds), Basic Processes in Reading, Perception, and Comprehension. Hillsdale, NJ U.S.A.: Erlbaum. SANFORD, A. J. and GARROD, S. C. (1981) Understanding Written Language. New York: Wiley. SCHANK, R. and ABELSON, R. (1977) Scripts, Plans, Goals and Understanding." An Enquiry Into Human Knowledge Structures. Hillsdale, NJ U.S.A.: Erlbaum. SCHANK. R., LEBOWITZ, M. and BIRNBAUM, L. (1980) An integrated understander. American Journal of Computational Linguistics 6, 13-30. SCHNOTZ, W. (1983) On the influence of text organization on learning outcomes. In Rickheit, G. and Bock, M. (eds), Psycholinguistic Studies in Language Processing. Berlin: de Gruyter. SHERRARD, C. A. (1985) Collapsing paragraphs: the engineer's writing problem? Paper presented at Meeting of the Cognitive Psychology Section, British Psychological Society, Oxford, September 1985. SHERRARD, C. A. (1986) Summary-writing: a topographical study. Written Communication 3, 25-30. SJOSTROM, C. L. and HARE, V.C. (1984) Teaching high school students to identify main ideas in expository text. Journal of Educational Research 78, 114-118. WINOGRAD, P. N. (1982) An examination of strategic differences in summarizing texts. Unpublished doctoral dissertation, University of Illinois, Urbana. WINOGRAD, P. N. (1984) Strategic difficulties in summarizing texts. Reading Research Quarterly 19, 404-425.