Early adolescent composing within a generative-evaluative computerized prompting framework

Cbnputer~ in Human l3ehavior. Vd. 8. pp. 3962.1992 Printed in the U.S.A. All rights reserved.

0747-5632192 $5.00 + .OO Copyright 8 1992 Peqamon Ress plc

Early Adolescent Composing Within a Generativ+Evaluative Computerized Prompting Framework Curtis J. Bunk West ~r~i~ia ~nive~i~

Thomas H. f?eynolds Universi(y of Wisconsin-Madison

Abstract - A review of cognitive process research on writing suggests that a generative-ev~l~tive model of ~o~osing could signi~cantly alter i~tru~tion on higher level writing skills and strategies. To test the model, a total of 164 children in the sixth, seventh, and eighth grades were blocked according to low and high writing ability and then randomly assigned to one of two groups: (a) a control group or (b) an experimental group which received computer prompts, think sheet scaffolds, and expert modeling of prompts. Results from dimensional and holistic assessments of written products indicated that students did not improve their writing performance as a result of exposure to prompts over a &week period. Internalization of writing-control strategies as measured by the Index of Writing Awareness (IWA; Bonk, Middleton, Reynolds, & Stead, IPPI), open-ended guidance questions in writing, and a prompt sort task all failed to favor the treatment group. In contrast, keystroke ~~lyses revealed that students in the tre~~nt group lowered the number of s~fa~e level changes they were bung per 100 words of text when the prompts were available, while at the same time increasing their textual repositioning within text. No diferences, however, were found in meaning changes as a result of these altered revisionary practices. As expected, though, students’ holistic scores in writing were significantly correlated with their awareness of writing skills and strategies. Surface changes were negatively correlated with students’ holistic scores, while repositioning within text correlated positively with holistic scores. A computer attitudes questionnaire indicated that the treatment group considered the computer helpful in creating a Portions of Association, Requests Educational

this paper were presented at the annuai meeting of the American Educational Research April 1990, Boston, MA. for reprints should be addressed to Curtis Jay Bonk, West Viginia University, Dept. of Psychology, 609 Allen Hall, Morgantown, WV 26506-6122. 39

40

Bonk and Reynolds

sense of audience and affording text evaluation opportunities. Various grade and ability differences provided evidence that procedural assistance was most appropriate at the sixth-grade level.

INTRODUCTION

Recent national concern over the decline in students’ writing abilities has compelled educators to seek fresh approaches to teaching writing skills. Educational theorists have moved beyond the perspective of writing as a simple, technical skill to recognize the possible linkage between writing and thinking skills and abilities (Applebee, 1982, 1984; Bereiter & Scardamalia, 1985; Flower, 1987). More spccifically, they propose that, just as higher order thinking involves the integration of convergent and divergent thinking strategies, effective writing is also aided by proficient use of generative and evaluative composition strategies (Flower dz Hayes, 1981; Scardamalia & Bereiter, 1985). This theoretical insight could significantly alter the way writing skills are taught. To date, however, there have been few assessments of the models and theories that detail writing as a generative/evaluative compositional process, or of the effectiveness of teaching strategies that stem from this theoretical perspective. To initiate evaluation of these models, some researchers have investigated the effectiveness of prompts on children’s writing (Bereiter dz Scardamalia, 1982; Daiute, 1985). In these studies, prompting strategies, termed procedural facifitators, allowed the learner to approach the writing task by using strategies that involved higher level aspects of their problem-solving processes. As yet, however, a theory of prompting for revision and/or generation of text does not exist. The goal of this research project was to build on prior work while attempting to develop and test a theory of prompted writing. The procedural facilitators used in the study were grouped by thinking skill; generative and evaluative prompts comprised the major classifications. Evaluative prompts, addressing the logical flow, relevancy, and consistency of ideas within and between statements, aimed to improve sentence clarity and the overall organization of the written text. Generative prompts, suggesting divergent new ideas and unique perspectives, encouraged the learner to creatively change directions and extend ideas and concepts. Researchers have used various methods for tracking the developmental change of students both within and between writing assignments (Bridwell, Nancarrow, & Ross, 1984). While Salomon (1988) and others (Woodruff, Bereiter, & Scardamalia, 1981) have investigated random and mandatory prompting treatments, respectively, the system proposed in this study allows the writer to choose a prompt during naturally occurring reflective moments in the composing process. The system encourages learner control in order to motivate students’ writing processes and foster the internalization of executive plans and goals within their writing. In addition, the system tracks student writing progress and revisionary practices through a keystroke-mapping program. THEORETICAL FRAMEWORK AND FOCUS OF THE STUDY

The design of this study is best described as a problem-solving approach to writing. The following researchers also have discussed this perspective with parallel terms and metaphors:

Adolescent composing within a generative-evaluative

computerizedframework

41

The kind of theory we envision views writing as a process of generating and editing text within a variety of constraints. (Collins 8z Genmer, 1980, p. 52) Writers and teachers have long argued that one learns through the act of writing itself, but it has been diffknlt to support this claim in other ways. However, if one studies the process by which a writer uses a goal to generate ideas, then consolidates those ideas and uses them to revise or regenerate new, more complex goals, one can see the learning process in action. (Flower & Hayes, 1981, p. 386) The above quotes are a reflection during the 1970s (Barritt Jz Kroll,

of the shift iu the field of Writing

that occurred

1978). At that time, predominant product views of writing gave way to cognitive process models. The product-oriented linear stage view of composing was replaced by a recursive model that identified planning, text generation, and revision as processes that took place at any moment in the writing process. One of the most inclusive and referenced cognitive perspectives on writing is the Flower and Hayes (1981) cognitive model of the composing process. The key assumption behind the Flower and Hayes model is that a writer does a great deal of thinking and problem solving at various levels of abstraction when composing. As the writing task or situation becomes more difficult, the writer needs additional problem-solving skills and strategies to manage it (Hayes 8z Flower, 1986). The planning, translating, and reviewing processes, and the organizing, goal setting, generating, evaluating, and editing subprocesses of this model are not distinct stages to be completed but are descriptions of the thinking processes that occur at any moment in the composing process (Flower Jz Hayes, 1981). This is a critical idea since it furnishes an extensive research agenda as well as a framework for cognitive psychologists to speculate about writing and reasoning. According to the Flower and Hayes model, the most obvious act of planning is the “generation” of ideas (Flower & Hayes, 1981). The writer’s plans can be in the form of pointers, markers, words, images, or goals (Hayes & Flower, 1986). The time spent planning for text generation is extremely critical to the quality of the final text. In fact, the research agenda of Flower and Hayes and their colleagues recently concentrated on the planning procedures of experienced and inexperienced writers (Carey, 1988; Haas, 1988; Higgins, Flower, & Petraglia, 1990). Collins and Gentner (1980) detailed ways to capture ideas, including: (a) creation of analogies, (b) formation of critical case comparisons, (c) definition of examples, (d) elaboration on dimensions, (e) recognition of opposite situations or dissimilar cases, and (f) simulation of a situation through overt actions or speculations. Yet, at some point during planning and idea generation, one’s ideas must be conceptually organized and revised (e.g., lists, categories, divisions, or concept maps), since conceptually oriented text generation enhances the caliber of plans and eventual quality of text (Scardamalia & Bereiter, 1982). For young writers, it is more demanding to verify that the text they have generated is translated into a coherent and meaningful whole than to generate propositions. The coherency check on ideas requires text evaluation, revision, and the generation of new plans. Efficient evaluation strategies operate not only on the actual text produced, but also on internal plans or pretext (Flower, Hayes, Carey, S&river, & Stratman, 1986). Thus, analysis of revisionary strategies is difficult, because experienced writers revise and reorganize not only surface or written text, but also ideas formed or in the process of being formed in their heads - pretext (Witte, 1985). The efficient writer’s task representation is not static at any point in com-

42

Bonk and Reynolds

posing, but changes through extended reinterpretations of plans and goals (Flower, 1987). In fact, Bereiter and Scardamalia advocated the use of the term reprocessing to extend notions of revision (Scardamalia & Bereiter, 1986). Reprocessing was seen as more appropriate since it accounted for whatever was produced in an episode (e.g., text, notes, new thoughts) that could be used in future processing cycles (Bonk & Reynolds, 1991). As a result, reprocessing does not add something new to the text but “transforms it” instead. Flower and Hayes (1981) also agreed that each word added to text or pretext constrains or limits future alternatives. They noted that writers consistently have a “re-seeing” or “re-visioning” of the text while composing. Writing is recursive, since both planning and revising go on throughout composing. As writers reflect on what they have written and attempt to rework their thoughts and ideas, they generate and regenerate goals and ideas. Hence, the discovery of a new idea can initiate revision somewhere in the actual text or in the pretext, and this revision can lead to further major or minor new ideas. But perhaps, even during idea production, evaluative processes are needed to decide if the new thought, concept, plan, or image is appropriate for the task. It is conceivable that the generative and evaluative subprocesses of writing are interdependent, even though they can be individually applied (Collins Jz Gentner, 1980). Collins and Gentner also suggested that the systematic switching between these two processes is what separates good from poor writers. In addition, Flower and Hayes (1981) underscored the fact that the subprocesses of generating and evaluating are somewhat unique - only these two processes can interrupt any other process at any time during composing. Or in their words: First, as our model of the writing process describes, the processes of generate and evaluate appear to have the power to interrupt the writer’s process at any point - and frequently they do. This means that new knowledge and/or some feature of the current text can interrupt the process at any time through the processes of generate and evaluate. This allows for a flexible collaboration among goals, knowledge, and text. Yet this collaboration often culminates in a revision of previous goals. (Flower & Hayes, p. 380)

This notion of an interaction between generative and evaluative processes in planning and revising is elaborated both in the following paragraphs and also in the discussion of the proposed model of the generative-evaluative cognitive processes in writing presented later in this paper (see Figure 1). Collins and Gentner (1980) noted that elementary students lack these important writing skills when they first attempt to write expository text because they are unable to concentrate or voluntarily interrupt their writing processes to focus on critical aspects of text. The findings from Collins and Gentner are interesting since they draw attention to the executive or monitoring component of the Hayes and Flower model. According to the Flower and Hayes (1981) model, the successful writer is one who is able to monitor and direct her own success. But what does this mean? As indicated, young children’s difficulties in generating and evaluating text lies in their lack of an executive routine to handle the switching between writing processes (Flower & Hayes, 1981). A high level executive routine is the most valuable component of text production, since it directs and regulates the whole writing process (Bereiter, 1980). Besides the regulation of processes, children need knowledge about the capabilities of their own thinking and regulatory processes (Flower, 1987). In effect, the monitor is useful both for the encouragement of new ideas and also for the evaluation of them. Even when children have concept generation and

Adolescent composing within a generative-evaluative

WWnq

Memory

3 General Coqmtivc 4. Task Farmliarity,

A.

tietacog.

B.

Cognitive

Some Control; Translating,

43

Cena!raint~

1.g-Termedge 2. Workinq

Reorocess~nq

computerizedframework

Capacity and Metacoqrutwe Skzlll Generality, end Complexity

5. Context: Genre, 6. Audience 7. 1ntereet 8. WntinqSklll

Setting,

Topx

Enwronment:

Level:

Level: Plsnmnq, & Revising

.

Planning-Orqamze Plane (Pretext or Text) _________________ Operation Sequence: Generate/Evaluate/Generate

C. Spontan.

I

Translate

Revise-Reorqan Plan ,~~Plene end .++(Text or Pretext) ____________________ Pretext to Operation Sequence: Words Evaluate/Generate/Evaluate

k@

Level:

lmuffic. Control; Random Text & Knowledge Telli “9

Planning-Content

d I 2. 3. 4.

DowelbAng ” from Movement toward Downsliding from Movement toward

reflective reflectlon reflective reflection

Generation

procettes toward automaticity or fluency. after prompted generative processes. processes to edit and make mechanical changes after prompted evaluative processes.

Figure 1. Proposed model of the generatlv~valuative

processes In composing.

rewriting skills, excessive demands of the writing situation inhibit their use (Flower et al., 1986). Moreover, monitoring changing plans, goals, and other constraints within the limits of current working memory is even more difficult for young writers (Flower, 1987). Therefore, those intervening in an inexperienced writer’s executive scheme or strategies must consider these limitations and adapt their treatments to them. Bereiter and Scardamalia (1985) suggested that elementary children have cognitive coping strategies that catapult them through tasks without the need for effortful higher order thinking or executive control processes. Children often do not have the benefit of feedback from a conversational partner when they enter the domain of writing (Bereiter, 1980). In effect, for the young writer to effectively communicate, she must adapt to the radical change from familiar and interactive conversational environments to solo writing environments. Without support, the text produced from young writers appears egocentric. According to Bereiter and Scardamalia (1985, 1987), young children’s writing is immature because they are operating within the “knowledge telling” problem space.

44

Bonk and Reynolds

Here children concentrate mainly on the words they want to say without framing them according to discourse conventions or audience expectations (Scardamalia, Bereiter, & Steinbach, 1984). One signal of text written in the knowledge-telling mode is content written in the order generated. Also, knowledge tellers will reexamine assignment cues for topical prompts when stuck, rather than inspect internal plans and goals (Scardamalia & Bereiter, 1982). As indicated by the generativeevaluative model of composing discussed later, the slightest prompting persuades knowledge tellers to write more since their major goal is to get content on paper or externalize preexisting knowledge (Bereiter Jz Scardamalia, 1985). Scardamalia and Bereiter (1982) posited an intermediate stage in writing development between knowledge telling and more mature “knowledge transforming.” In this stage, the young writer depends on concentrated self-cuing or outside intervention in order to generate extended written discourse while using conversation-related schema structures. But initially, the mechanical constraints of text production drain available cognitive resources, eliminating opportunities for self-questioning or internal dialogues to develop (Bereiter Jz Scardamalia, 1987). A number of researchers claim that new computer-based writing tools can reduce working memory constraints by functioning as collaborative writing partners that explicitly display complex cognitive operations inherent in writing (Glynn, Oaks, Mattocks, & Britton, 1989; Kellogg, 1989; Kozma, in press; Montague, 1990; Salomon, 1988; Smith & Lansman, 1989; Zellermayer, Salomon, Globerson, & Givon, 1991); however, for this to occur many existing software packages may first require minor repurposing (Bonk & Reynolds, 1991). Basically, these researchers argue that for young children to consider rhetorical constraints, such as audience, in a similar fashion to the cues and constraints provided by a conversational partner, they need computer and other support strategies. Recently, McCutchen (1988) made a critical distinction between automaticity (i.e., fluent subprocesses that are unregulated or unchecked) and functional automaticity (fluent subprocesses that are strategically guided) operating within writing. She concluded that the major difference between knowledge tellers and skilled knowledge transformers was in their metacognitive control over their writing fluency subprocesses (i.e., their functional automaticity). In addition, she argued that highly automatized procedures can be costly since they are resistant to intervention and difftcult for the writer, teacher, or researcher to analyze. McCutchen proposed that writing-skill interventions must attempt to strike a balance between fostering automaticity and encouraging metacognitive awareness and control of the writing process. She suggested that a mild form of intervention such as procedural facilitation (described below) may be one way to overcome common “content spills,” while fostering metacognition in writing. Procedural facilitation provides young writers with temporary supports, similar to those they previously received in conversation, thereby encouraging them to adopt more complex writing strategies. Of importance to this study, McCutchen argued that using procedural facilitation in disciplines where rule systems are too complex to teach (e.g., writing) would promote self-regulatory mechanisms (Bereiter & Scardamalia, 1987). This type of intervention technique oversees the executive monitoring system, while providing cues or routines for switching the writer between text generation and revision. Scardamalia and Bereiter (1985, p. 566) defined procedural facilitation as “routines and external aids designed to reduce the processing burden involved in bringing additional regulatory mechanisms into use.” Basically, procedural facilitation was designed as a tool for developing children’s metacognition in writing (Bereiter & Scardamalia, 1987).

Adolescent composing within a generative-evaluative computerizedframework

45

One of the key goals of the prompts is to help the child become more reflective when approaching a writing task. Procedural assistance can offer enco~gem~t in (a) generating content, (b) keeping the student on topic, (c) generating approp~a~ plans and goals, (d) dia~os~g and operating on textual ~consistencies and problems, and (e) producing a coherent whole (Scardamalia & Bereiter, 1985). In effect, covert mental operations are made overt. Procedural facilitation is contrasted with substantive facilitation, in which outside support actually reduces part of the task burden or intellectual demands of the task. In substantive facilitation, comments or cues are directed specifically at something the student has written (i.e., they are topic-specific), whereas in procedural facilitation the cues are extremely general and nonspecific but, nonetheless, familiar to the writer. The prompts used in procedural facilitation techniques act as a partner would in a conversation, enabling the writer to decenter from personal thoughts and thii about potential readers. This awareness of the reader, or soci~-co~itive ability, may be one of the most difficult and vital aspects of effective writing (Bonk, 1990, Martlew, 1983). By forcing the writer to reflect on her writing routines, the prompts may encourage automated processes to come under review and eventual control (McCutchen, 1988). What the technique essentially facilitates is an internal dialogue, or at least some form of assisted monologue (Daiute, 1985; Scardamalia et al., 1984). When made salient through procedural facilitation, the conflict between one’s actual and intended texts is an invitation for reflection. Pausing to internally reflect and discuss textual inconsistencies or audience needs is a valuable writing skill (Martlew, 1983). However, the internal dialogue of the expert is hard to find in the verbal protocols of novices (Daiute, 1985; Scardamalia et al., 1984). Therefore, the intent of signals sent by external sources such as procedural prompts is to encourage the child to work within an externally supported, interactive writing enviro~ent that eventually functions as an independent inner dialogue. To initiate evaluation of interactive models of the composing process, some researchers have investigated the effectiveness of prompts on children’s writing (Daiute, 1985; Scardamalia & Bereiter, 1982). They have given students sentence openers (e.g., “I think. . .” or “One reason is. . .“), contentless prompts (e.g., “Can you write some more?“), and planning and goal-setting prompts in cue card and computer support formats (e.g., “Give an opinion”) (Scardamalia & Bereiter, 1986; Woodruff et al., 1981). As yet, however, a complete theory of prompting for generation, revision, or reprocessing of text does not exist. In addition, though researchers allude to the importance of switching between generative and evaluative aspects of text, procedural facilitation research has yet to incorporate both generative and evaluative processing support. Research from the Flower and Hayes group has addressed planning (Carey, 1988; Haas 1988), while others have focused mainly on evaluative and revisionary procedures in writing (Bereiter & Scardamalia, 1987; Daiute 1985, 1986; Scardamalia et al., 1984; Woodruff et al., 1981); no one has investigated the combined effects of generative and evaluative procedural facilitation. In addition, limited theoretical grounding, narrow prompt focus, and brief treatment length (e.g., regarding length, see Woodruff et al., 1981) confound the results of many studies on procedural facilitation in writing. Hence, no study has extensively compared prompt effectiveness across grade and ability levels or incorporated subst~tial sample sizes (i.e., 100 or more subjects). (For a review of preliminary findings regarding procedural facilitation systems [prompted writing] see Bonk, 1989.)

46

Bonk and Reynolds

Bereiter and Scardamalia (1987) argued that procedural facilitation was one method to lead students on a path of active, self-regulated learning. When students are stuck or inactive, they can use the procedural prompts to get more involved in the activity. Thus, there must be conscious and purposeful use of this strategy. Additionally, the more students understand of the strategy (e.g., through expert modeling and peer support), the more likely they will feel a sense of ownership in it and be motivated to use it (Freedman, 1987). Finally, in order to maximize the instructional effectiveness of the intervention, procedural facilitation is aimed at the upper limits of the child’s learning envelope or the growing edge of competence (Bonk & Reynolds, 1991; Vygotsky, 1986). A MODEL OF GENERATING AND EVALUATING TEXT

According to Caccamise (1987), planning is the particular writing process that should be the main concern of researchers. As alluded to earlier, this is supported by the fact that 80% of composing time is typically spent planning. Caccamise’s own investigation into planning and idea generation has demonstrated that when a student receives or creates a topic, a retrieval probe is generated to search preexisting world knowledge of the topic. At the same time, an immediate editing process develops according to task familiarity. The more familiar the topic, the greater the number of ideas generated. The size of the search set also depends on the task specificity: The more general the probe, the greater the number of possible concepts and ideas generated (though these ideas may lack tight organization). Nevertheless, planning involves both the generation and evaluation of ideas before, during, and after text transcription. There is a profound difference between the generation strategies of mature and immature writers. When experts produce an idea, they elaborate on it, develop it in greater depth, and connect it to other ideas in pretextual or textual form (Caccamise, 1987). The fiial product of idea generation probes is a writing plan that vaguely resembles the actual text produced. In contrast, immature writers engage in loosely structured knowledge telling. Children who fail to distinguish between content generation and transcription need an external stimulus to disrupt their continuous text generation processes, thereby refocusing them on conceptual plans and goals (Bereiter & Scardamalia, 1987). In comparison to idea generation, a similar pattern holds for revision. Revision is a central part of writing (Fitzgerald, 1987) that extends considerably beyond mechanical editing (Bartlett, 1982). Revision must be viewed as a separate process from translation, since it involves comparing actual text with intended text and operating on those differences. Because revision often leads to new thoughts and ideas, it is seen as vital to the generative plans and goals of writing. Thus, revision also involves both generative and evaluative processes. Moreover, just as Caccamise (1987) pointed out that there are numerous constraints during planning and idea generation, similar constraints and situational variables affect revision and resultant writing quality (e.g., genre, task familiarity, subject, interest, length) (Witte & Faigley, 1981). There are many differences already noted in the revising strategies employed by experts and novices. Experts make more revisionary changes that affect the meaning of the text, while inexperienced writers focus on the local level of textual revi-

Adolescent composing within a generative-evaluative computerizedframework

47

sion characterized by spelling, word, or other surface level changes (Witte & Faigley, 1981). Further, while experts wait until after topical criteria have been satisfied to correct most mech~i~l errors, ~x~rien~ writers often stop writing and reevaluating the gist of their text in order to correct minor errors, but not meaning, As with planning, novices must be made aware of larger textual and rhetorical revisionary concerns. As noted in the author’s model displayed in Figure 1, both generative and evaluative processes are critical to planning and revising text and pretext, though in different strengths. Note that the emphasis remains on idea generation in the planning phase and on the evaluation of those ideas in the revision phase. However, metacognitive control is needed for the evaluative processes to have a significant role in text generation sequences and for revision to be seen as a generative process; uncontrolled processes can have destructive and uncertain consequences in writing (McCutchen, 1988). In Figure 1, writers operating at the lower (spontaneous) level are generat~g text according to content goals or length requ~emen~ without an interaction occur&g between generative and evtiuative processes in either their plans or revisionary tactics. Procedural prompts perform a critical function in refocusing students’ attention toward the interaction between generative and evaluative processes. Students should be most concerned with the coherence, consistency, elaborateness, and logical sequence of their ideas, not the mechanics (e.g., spelling) (Bereiter & Scardamalia, 1987). They must be made aware of the interconnectedness of their thoughts; each change made within either extant text or pretext impacts on plans for every other thought, image, or note. As mentioned earlier and noted in the generative-evaluative model of writing, the ideal textual environment is one of reprocessing (Bonk & Reynolds, 1991; Fitzgerald, 1987; Sc~d~~ia & Bereiter, 1986), not spon~eity. When elementary students are permitted to operate in the spontaneous level of knowledge telling, they “downslide” or get pulled into the local demands of topicdirected content generation and/or editing for mechanical mistakes (Bruce et al., 1982). Interventional tactics using computer tools might refocus children’s attention to a more “principled” or metacognitive level (Bonk & Reynolds, 1991; Comeaux & Lehrer, 1986), perhaps the intermediate one Scardamalia and Bereiter (1982) claimed existed between knowledge telling and knowledge transforming. In the model presented in Figure I, generative and evaluative prompts are noted as one way to shift children’s attention to higher levels of processing and control in the writing hierarchy. In this model, me~~o~itively aware writers organize and reorganize textual and pretextual plans and goals. Thus, ~~rn~i~tion of generative and evaluative cognitive processes and procedures of composing enables the knowledge teller to move toward a more mature stage of writing. Editing and content generation are not entirely neglected processes in this model; they are still available when needed. For example, the writer may, at times, have thoughts flowing so fluently that she would want to freely write them down without the additional demands of evaluation. At some point, however, she would come back to analyze these thoughts for overall coherence. Thus, the mature writer who experiences spontaneous text generation or “stream of consciousness” maintains some degree of deliberateness or control over actual text produced McCutchen’s ( 1988) notion of functional automati~ity. The proposed model indicates that there are hier~~hical levels within writing that function together in a “writ~g-as-repr~ess~g” framework to enhance writing

48

Bonk and Reynolds

quality. (See Bonk & Reynolds, 1991 for elaborations on a reprocessing framework for writing, especially in regard to computer tools). The generative and evaluative organizing and reorganizing of one’s plans and goals is the critical difference noted in this model of writing. 1 Without deliberate attempts to generate and evaluate plans and text, writing would not exist. Yet, as is evident throughout the literature review, this interconnection between generative and evaluative processes is often absent in children’s writing. Our study attempted to repair the missing processes by exploring the effectiveness of computerized generative and evaluative prompts on the development of children’s writing. Development was explored in a number of ways. First, midtreatment and posttreatment writing quality data were compared between the treatment group (prompted condition) and the control group. Second, development was analyzed between three grade levels (sixth, seventh, and eighth) and between high and low ability students. Third, development within an individual writing assignment was tracked and reevaluated through a keystroke mapping system. Fourth, possible internalization of prompts was examined. Finally, children’s writing-gain scores over a IO-week period were compared.

METHOD

Subjects

A total of 164 middle school students (53 sixth-, 52 seventh-, and 59 eighth-grade students at a rural midwestem school within communicating distance of a major urban area2 took part in a 13-week course (one session/week) on electronic writing using a common word processing package.3 Thus, this study attempted to address treatment length, grade level, and sample size inadequacies of previous research. The next section details a procedural facilitation system employed to: (a) integrate typical prompt categories found in the literature and (b) test the model of generativeevaluative processes of composing. Software Materials The procedural facilitators used in the study were grouped by thinking-skill based

on the cognitive process model of Flower and Hayes (1981) and the generative-evaluative model of writing explained previously; not surprisingly, generative and evaluative prompts comprised the major classifications (see Table 1). Generative prompts suggesting divergent new ideas and unique perspectives were developed to encourage the writer to creatively change directions and extend ideas and concepts. Four subcategories created for each main category gave students the means to conceptualize these questions and use them. The four generative prompt categories were derived from the literature on creativity (Davis, 1986; Torrance, 1974) and were as follows: (a) fluency (the ability to produce many types of ideas), (b) flexibility (the ability to take different approaches to a problem or think of different categories), (c) originality (uniqueness or nonconformity in thought), and (d) elaboration (the ability to add details or embellishments to an idea) (Davis, 1986). These same categories were subscores of the widely used Torrance Tests of Creative Thinking (Torrance, 1974). Due to the abstractness of these terms, the names were scaffolded down to the appropriate ability level. Subjects knew the generative prompts as “more ideas,” “types of ideas,” “new ideas,” and “extenders.”

Adolescent composing within a generative-evaluative computerizedframework

49

Table 1. Generative and Evaluative Prompt Listing Generative Prompts fluency (Mkkfb school students knew this as: MORE IDEAS): Q: List all that you know about your topic in your head or on paper. You may want to jot down items in the list that are not in your paper. A: Ask yourself: What other ideas does this suggest? What could I add here? And, how could I exaggerate or maybe say the opposite? 2: What else might your audience want to know? Would the reader want to know about the smell, sight, sound, or touch of your object? Flexibility (TYPES OF IDEAS): W: Add other categories, models, examples, or lists. You might try to use pictures in your head to compare points. S: Just imagine if everything you’ve said so far is wrong. If the reader caught it, what changes might he/she suggest? X: Think again about your reader. Are there other points of view that are necessary for your reader to understand? Ordinally (NEW IDEAS): ._ E: Try out a wild idea or describe your last thought in a metaphor. How is a _ lie a ~. 373 D: Ask yourself ‘What if...?” and then reflect on what might happen to change your mind on this topic. C: Try combining two or more of your ideas into something really unique. Have you used your creativity or imagination? Elaboration (EXTENDERS): R: Have some fun, play with the last idea, expand or extend it, and then maybe contrast it with something else. F: Reread your last paragraph. Would expanding or adding a sentence help your reader understand? V: Try to broaden the focus of your paper by including exceptions to what you are saying. Evaluative Prompts Relevancy (QUALITY): you’ve said needed or related to it? T: Think about the problem or orfginal topic. Is eve~hi~ G: Reread your paper and delete repeated or unneeded sentences which don’t he$ form an overall theme. _ . . ES: Try to see or imagine where your writing is headed. Is the informat~n you’re providing good and also relevant? Logic (CLEAR/LOGICAL): Y: Give an example that might make your reasoning clearer to the reader. State all examples in clear and simple ways. H: Read the first and last sentences to each paragraph. Are there transitions from one sentence to the next? N: Think back about your original idea or opinion on this topic. What can you say now to provide support for your entire paper? Assumptions (ASSUMING): U: Reflect on the sources of your information. Are your sources and your assumptions stated as such in your paper? J: Read over your paper for personal bias; look out for sentences where you say “I feel” or “I think” w~~ut backing them up. M: Will your audience agree with your values, opinions, or ideas? If not, list something that might help get your point across. Conclusions (CONCLUSIONS): I: Have you provided enough information to back up your claims and conclusions? And are there other effects to what you’re saying? K: Are there diierent conclusions to what you are saying? Try to explain these so they make sense for the reader. 0: Can you summarize to the reader what you have said in one or two sentences? Try to do this at the end of each paragraph or idea. Other Prompts (for whole paper): L: Step back and took at your whole paper. Are your thoughts and ideas logically stated, justified, interesting, and unique? P: (Clears the bottom screen) Note. The letter displayed (e.g., Q) is where that particular prompt was located on the keyboard.

50

Bonk and Reynolds

Evaluative prompts, which addressed the logical flow, relevancy, and consistency of ideas within and between statements, aimed to improve sentence clarity and the overall organization of the written text. The evaluative prompt categories were as follows: (a) relevancy (needed information that enhances the overall theme), (b) logic (clear flow of the paper), (c) assumptions (recognizing bias in one’s thoughts), and (d) conclusions (drawing appropriate summary statements). These categories were derived from overlapping categories within prior procedural facilitation revision studies by Daiute (1985) and Scardamalia et al. (1984), and various critical thinking projects involving writing (Talbot, 1986). Students understood the four evaluative categories as “quality,” clear/logical,” “assuming,” and “conclusions.” Pilot testing with 4 sixth, seventh, and eighth graders and four college students assessed prompt effectiveness. Considering the categories and the type of assignments they were designed for, it was not surprising that the prompts were more facilitative for expository text than for narratives. Prompts rated highly beneficial by subjects in the pilot research were selected for this study. Prompts were organized from left (generative) to right (evaluative) in a matrix on a computer keyboard (see Table 1 for sample prompts). Three keys running diagonally down the keyboard formed one category (e.g., the q, a, and z keys formed the fluency category). The prompts were in the form of statements and questions requesting that the writer ponder over possible textual changes (e.g., “Have you provided enough information to back up your claims and conclusions?” or “Try to broaden the focus of your paper by including exceptions to what you are saying”). The prompts were programmed into WordPerfect’s macrofunction capabilities. When AltShi&letter combination keys were pressed, a prompt appeared in the bottom 2 lines of the screen (22 lines of text were saved for the paper). At that point, the student had the option of either using the prompt for reflection, continuing with current thoughts, or invoking another prompt. Meaning and surface changes as a result of the prompts were recorded through a keystroke mapping system described later. Procedure In the initial 3 weeks (or three sessions), students were given preliminary training

on WordPerfect during November 1988. After preliminary training, they were blocked according to low and high writing ability and then randomly assigned within each grade to one of two groups: (a) a control group or (b) an experimental group which received computer prompts, think-sheet scaffolds, and expert modeling of prompts. The experimental group viewed a 20-min videotape of the two expert college writers, one male and one female, which showed how to use the prompts. Next, there was a live 5-min modeling session of the computerized prompting program. After the modeling, students were given a chance to practice the Alt-S/z@-letter combination needed to invoke a prompt. Finally, the experimental group received a list of the generative and evaluative prompts for referential and recording purposes as an additional mediational feature or instructional strategy scaffold (see Table 1). Students in the experimental groups were asked to use the prompts at least 12 and 10 times, respectively, during the two sessions of Paper 2; 8 and 6 times, respectively, during the two sessions of Paper 3; and when needed during the two sessions of Paper 4 - the final exposure to the prompts. More specifically, they were asked to use them during reflective moments in the composing process. Of the 10 weeks of word processing in the spring occurring after the 3week training period, the prompts were available only for Weeks 3-8.

Adolescent composing within a generative-evaluative computerizedframework

51

A blocking variable on writing ability was incorporated into the design based on the average of each student’s reading and language scores from their Metropolitan Achievement Test results. The blocking variable was added to determine the effects of prompts on low and high ability students. Students composed their expository texts in two sessions. Topics were assigned one week before actual composing time to allow students time to prepare according to their preferred planning strategies. Before starting each writing session, students were told to concentrate on their content, rather than spelling or grammar. Fortyfive minutes was allocated for each class, though only 40 min was available for composing, due to experimenter set up time. After the 3 weeks of training, the remaining lo-week writing project consisted of three analyzable events. First of all, during the initial 2 weeks of training in January, Writing Task 1 served as baseline data. During the next 6 weeks (Weeks 3-8), students in the experimental group had the prompts available during composing (Tasks 2-4). The final task (5) was assigned during Weeks 9 and 10 without the availability of the prompting program. Therefore, Task 5 was used as a posttest of writing ability. The pretreatment, midtreatment, and posttreatment data were used to make comparisons of the writing samples over the lo-week intervention and between groups, grades, and ability levels. Expository writing assignments were chosen for all five assignments since they required generative/inventive cognitive processes (Ruth & Murphy, 1988), as well as advanced evaluative/revisionary processes (Bereiter & Scardamalia, 1987). The five writing tasks were selected according to student interest across grade levels and gender. Instruments In similarity to Bridwell and colleagues (Bridwell et al., 1984; Bridwell, Sire, &

Brooke, 1985), a keystroke mapping procedure was used to track revision from multiple writing samples for every writer within each task. The keystroke mapping tool was designed to replay individual keystrokes at preselected speeds,4 thereby providing a chronological picture of student drafts and detailed documentation of within-draft changes and individual progress (for more information on this procedure, see Reynolds Jz Bonk, 1990b). Instead of analyzing these writing tasks within a “draft-and-revise” scheme, each paper was analyzed using the “reprocessing” framework or as one semantic unit from beginning to end (Bonk & Reynolds, 1991). Using a modification of Bridwell’s revisionary classification scheme (Bridwell, Sire, & Brooke, 1985), the replay of keystrokes was primarily focused on the following revisionary features: (a) surface revisions (a collapsed category including formal [format, spelling, grammar changes] and meaning-preserving [word changes]) and (b) meaning revisions (text-based changes at the phrase, sentence, and multisentence level) (Faigley & Witte, 1981). In addition, errant keystrokes were added to the original revision taxonomy to separate typo- or keyboard-related revisions from standard spelling, format, grammar, and word changes. An errant keystroke was considered to be any format or spelling change within a word that took place before starting another word. Prompt effectiveness also was recorded. Since writing involves many dimensions besides revision, there was a need to combine the keystroke analysis with other research tools. Therefore, the evaluations of the written products extended beyond analyses of in-process revisions to both holistic and analytic assessments. When considering the generative-evaluative focus of the present study, the Analytical Scheme for Critical Thinking Skill

52

Bonk and Reynolds

(recommended by Hawisher & Fortune, 1988, for investigation of writing and reasoning skills) was ideal (see Purves, 1985, for details). The original instrument is composed of three dimensions: (a) content thinking (seven items), (b) organization (three items), and (c) style/tone (three items). To further evaluate the effectiveness of our prompts, the content-thinking and organization components of Purves’ dimensional scoring scheme were reordered into generative and evaluative subscales. Items 1 (adequacy of information), 2 (the richness of information), 4 (the number and depth of inferences made), and 7 (consideration of alternatives) comprised the content-thinking-generative subscale. Items 3 (relationships drawn), 5 (synthesis or drawing together of ideas), 6 (evaluation or making judgments as to the merits of ideas), and a combination of Items 8-10 (framing, grouping, unity) comprised the content-thinking-evaluative subscale. The 495 papers (3 [pretest, midtreatment, and posttest writing samples] x 165 subjects) in this study were rated using all eight dimensions on a scale of 1 (low on this category) to 5 (high on this category). The modification of Purves’ critical thinking instrument for the purposes of this study is listed in Table 2. Recently, Hawisher and Fortune (1988) found that a holistic scoring framework facilitated later dimensional evaluation using Purves’ critical thiiing scheme. A holistic rating also provided an overall impression of composition quality that Purves’ instrument lacked. The current study followed Hawisher and Fortune’s (1988) lead in using Purves’ thinking-skill measure in conjunction with a holistic instrument to judge writing quality. The holistic instrument used in this study was developed by the Educational Testing Service (ETS, 1984). The scoring was based on a 6-point scheme. One week after the treatment period, students completed five different instruments: (a) a metacognitive measure which incorporated three open-ended questions regarding the giving of advice in writing (Englert, Raphael, Fear, & Anderson, 1988; Salomon, 1988); (b) one open-ended question regarding the giving of advice in reading; (c) a questionnaire regarding students’ perception of the assistance provided by the computer system; (d) the Index of Writing Awareness (IWA) (see Bonk, 1989); and (5) a 13-item prompt-sort task. These instruments were administered to all students to assess the internalization of the questioning strategies and heuristics. The prompt-sort task required students to select questions or prompts which appeared to be similar to each other. In this task, 2 points were allotted to within-category responses (e.g., both addressing fluency), 1 point for strictly generative or strictly evaluative responses, and 0 points for responses which mixed generative and evaluative prompts. The IO-item computer questionnaire was designed to learn whether the computer was regarded as a conversational partner or tool within any grade or group. Finally, the IWA was based on the Index of Reading Awareness (IRA) which was developed to measure children’s metacognition about their reading skills (Jacobs dz Paris, 1987). Bonk, Middleton, Reynolds, and Stead (1991) discussed the usefulness of the IWA in diagnosing early adolescent writing difficulties along with various instrument reliability issues.

RESULTS

Both the metacognitive advice scores and the holistic and dimensional ratings of the papers required interrater reliability checks. All raters were trained using papers

Adolescent composing withina generative-evaluativecomputerizedframework

53

Table 2. Adaptation of Purws’ Dimensional Scoring Instrument Dimensional Scoring Soiing Guide: Rank each composftiin on each aspect below on a scale from 1 (low in thii Gary) to 5 (displays this quality in a highly appropriate fashion). A 3 would indicate adequate performance. Rank each aspect separately. Generative Criteria 1. Adequacy of information presented - enough to fulfill assignment. Refers to the degree to which all of the relevant information from the stimulus is contained in the text (similar to Fluency - more ideas). 2. Considerationof afternatives - other arguments or interpretations and opinions of others. Refers to the extent to which the writer appears to admit the possibility of alternative or counter arguments or interpretations and accepts them as admissible or rebuts them. Includes implied rebuttals, qualifying language, hedging, modesty, and consideration of other viewpoints (similar to Flexibility -thinking of different approaches or categories). 3. Inferences made beyond the scope of the assignment. Refers to the number and depth of inte~retations (casual, resultant, comparative, contrastive, extrapolative) that the writer makes beyond the info~t~n in the stimulus and or from the outside (similar to Or~inali~ - uniqueness or no~nfo~~ in thought; see also Elaboration). 4. Richness of additional information - relevant allusion (an implied or indirect reference). Refers to the use of additional information/details to that which is in the stimulus (e.g., information drawn from a variety of sources such as reading or general knowledge) and may be seen as the amount of relevant allusion (similar to Elaboration - adding details and embellishments). Evaluative Criteria 1. Relarionships drawn among items of information. Refers to the degree to which the text shows that connections have been made between the various items of information and the validity and/or complexity of the relationships drawn (similar to Relevancy - needed information that enhances overall theme; see also Logic). 2. Framin~Grouping/Unity. Refers to the degree to which the writer presents the content in such a manner that there is an apparent beginni~, middle, and end to both paragraphs and the paper. It also refers to the degree to which the writer joins the various pieces of info~a~~n, rela~~nships, or inferences into paragraphs or some other means. Inadequate writing would be denoted by extraneous matter and thoughts, combining bits of ~nformat.~n, and lists with no discernible pattern or system (similar to Logic - clear flow of the paper; see also Relevancy). 3. Evakfation - the making of judgments as to the merit of ideas. Refers to the degree to which the wriier appears to make judgments as to the relative merit of particular relationships, inferences, or synthesis and the degree to which applicable criteiia are used. Includes implied and explicit reasons why suggest something, why change something, why build something, and why qualified to do something (similar to Assumptions - recognizing bias in one’s own thoughts; see also Relevancy). 4. Synthesis - drawing together ideas into a generalization. Refers to the degree to which the writer appears to draw together information, relationships, and inferences into a single or complex generalization (similar to Conclusions -drawing together appropriate summary statements). /Vote. Adapted from Purves (1985). Permission granted by the author to modify and reproduce.

not included in the final assessments (e.g, Tasks 2 and 4). A stratified random sample of the open-ended advice questions in reading and writing were scored by a second individual. Interrater correlations were based on a sample of 30 cases out of a possible 164 cases. Interrater correlations were r = .91 (p c .Ol) on the openended question in writing, and r = .85 (p c .Ol) on the open-ended question in reading. Based on an .80 criterion these coefficients were considered acceptable. Ninety (about 20%) of the total 495 essays within Papers 1, 3, and 5 were randomly selected and divided equally between two local writing experts. Stratified random sampling ensured an equal number of subjects selected by grade, group, and ability. None of the correlation scores met criterion. As a result, all papers were blindly reread and restored; discrepancies from the first scoring were recorded. At that time, a third rater, who received more training than the first two, was hired to blindly score a new stratified sample of 90 papers (20% of the total) - 30 from

54

Bonk and Reynolds

each assignment, once again equally split between grade, group, and ability. The correlation coefficient between the holistic scores of Rater 1 and Rater 3 and the experimenter was .83. Interrater correlations across all eight dimensions of the Purves instrument was .73. These correlations were considered acceptable. In the primary analysis, it was expected that the treatment group would perform qualitatively better than the control group on the midtreatment (with prompts) and posttreatment (without prompts) papers. Group by ability differences was expected to favor the treatment subgroup within each grade. This hypothesis was tested using the holistic, generative, and evaluative scores of Papers 3 and 5. Planned comparisons were performed within each grade and ability level to compare specific groups. Six specific planned comparisons were performed on each of these scores. The comparisons were designed to investigate possible treatment (experimental vs. control) differences within the low- or high-ability students at the sixth, seventh, and eighth grades. The acceptable alpha level was set at .05 for all planned and post hoc comparisons. Comparisons were performed for both Paper 3 (the prompted condition) and Paper 5 (the posttest without prompts). Dunn directional planned comparisons were calculated on this data. Only one comparison was significant. On the posttest, the sixth-grade high-ability treatment group outperformed the corresponding control group on both the holistic scoring (t = 2.15, p < .05) and the evaluation dimension of the modified Purves instrument (t = 2.54, p c .05) (see Bonk, 1989 for additional data regarding either Paper 1 or 3). The next hypothesis addressed students’ writing development over time. This hypothesis stated that, over the course of 6 weeks, students in the experimental group would internalize the generative and evaluative prompts. Internalization was expected to be reflected in higher holistic and dimensional treatment change scores by each ability subgroup compared to the control subgroups. In order to determine whether any of the six treatment subgroups improved over the prompting period, pretest to posttest change (gain) scores were obtained for the holistic, generative, and evaluative measures. Change scores were derived by subtracting the pretest score (Paper 1) from the posttest score (Paper 5) for each subject. The identical sixsubgroup Dunn directional planned comparisons were performed on this data as for the fist hypothesis. None of the planned comparisons within the 12 subgroups were significant. Additional questions regarding the internalization, transfer, and usefulness of the prompts were expected to be answered by administering (a) the TWA, (b) the sort task, the open-ended metacognitive guidance questions in (c) reading and (d) writing, and (e) a computer attitudes questionnaire at the end of the treatment period. In similarity to the previous two hypotheses, it was predicted that the six treatment subgroups would score higher than their matched control groups on each of these five measures. As noted earlier, each of these instruments was designed for a specific purpose. For example, the sort task was intended to find out whether there was a difference in students’ internal categorization of the prompts, while the openended question regarding guidance in reading was included to test for possible transfer of what was learned to reading. Pairwise comparisons for the three internalization scores - the sort task, the TWA, and the open-ended question measuring metacognition in writing - were performed (see Bonk, 1989, for comparison data on the other two measures: the transfer task to reading and the attitudes questionnaire). Results repeat the basic findings of the previous two hypotheses; even with directional (one-tail) hypothe-

Adolescent composing within a generative-evaluative computerizedframework

55

ses, only one of the pairwise comparisons was significant. The seventh-grade highability treatment group scored significantly higher than its control group on the open-ended question regarding advice they would give a friend who did not read well (the transfer task): t = 2.11, p < .05. Though both the low- and high-ability treatment sixth graders outscored their control counterparts on all five of these measures, the differences were not significant. Students in the treatment group were predicted to increase the number of repositions and meaningful revisions within their papers while writing Paper 3 (the prompted condition). In addition, because the computerized prompts were expected to focus the treatment group primarily on generating and evaluating content, it was postulated that they would make less surface changes per 100 words of content generated as compared to the control condition. Since a limited number of papers were keystroke analyzed (60 total papers), Grade x Group planned comparisons were performed to increase the power of the analyses, instead of the Grade x Group by ability comparisons that were performed on previous measures. Due to disk failures in the keystroke-mapping files near the end of the study, only Paper 1 (pretreatment) and Paper 3 (midtreatment) were available for keystroke analysis; hence, difference scores were obtained for these revision-related data. Difference scores between the pretest (Paper 1) and the prompted condition (Paper 3) were created for (a) surface changes (i.e., grammar, format, spelling, and word changes), (b) meaningful changes (i.e., phrase, sentence, and multisentence changes), and (c) page repositions/scrolling within a paper, No differences were predicted for the fourth measure, errant keystrokes, as a result of the treatment. The difference scores for each measure were used to perform treatment versus control planned comparisons at each grade level. Though none of the original Dunn directional Grade x Group comparisons were significant, the mean surface revision scores for the treatment group within each grade were favorable (lower) when compared to their respective controls. Hence, it seemed that the treatment subjects reduced the number of surface changes they made when writing in the prompted condition as compared to both their pretest scores and the control group, although again this was not statistically significant. Moreover, the treatment subjects appeared to have increased their amount of textual scrolling or repositions as compared to both their pretest scores and the control group. In order to increase the power of the calculations and investigate possible global group differences (instead of Grade x Group) for each of these measures, the appropriate 2 (Groups) x 3 (Grades) analyses of variance (ANOVAs) were executed. The ANOVA for surface changes confiied that there was a main effect for group: F(1, 48) = 5.88, p c .05. The treatment group did make less surface-level changes per 100 words of text produced when exposed to the prompts than during the pretest (Paper 1) (treatment: X = -2.89, SD = 4.05; control: X = -.35, SD = 3.961. The ANOVA for reposition changes failed to find a main effect for group: F(l, 48) = 3.11, p > .05. However, because this test approached significance (p < .lO), an additional post hoc ANOVA was performed on the reposition scores within Paper 3 only). Here, a main effect for group was found: F(l, 48) = 7.77, p < .05. The treatment group repositioned within their texts more than the control group when they were in the prompted condition (treatment: X = 5.76 repositions, SD = 8.70; control: X = 3.77, SD = 5.62). In addition, a two-way interaction between group and ability was detected by this ANOVA: F( 1,48) = 5.67. The significant interaction occurred because high-ability treatment subjects appeared to respond to the prompts by scrolling through their texts more during the prompted

56

Bank and Reynolds

third paper than low-ability latent students (high ability: X = 15.4, SD = 12.73; low ability: X = 5.47; SD = 6.85). In contrast, low-ability control students scrolled through their texts slightly more often than high-ability students, though for neither of these control groups did scrolling exceed 5 times per paper. Note that there were a number of other grade and ability differences that were not the primary focus of this intervention and are only briefly reported here (see Bonk, 1989, for more details). For instance, on the holistic, dimensional, metacognitive, and revisionary scoring, high-ability students significantly outperformed the low-ability students. Though this validates the use of the Metropolitan Achievement Test (MAT) scores as a blocking measure, there also is a developmental si~i~c~ce in this fmding. If a causal connection were established between one or more process and product variables, addition direct ~s~ction~ strategies and ~te~entions could be developed. In terms of grade differences among these measures, the seventh and eighth graders significantly outperformed the sixth graders on most measures as well (including the IWA, dimensional, and holistic scores). There was a consistent jump in performance and metacognitive awareness after sixth grade; this discovery could help situate instruction at the age where it is most appropriate. In addition, documented grade differences in metacognitive awareness or revisionary practices might help explain grade level shifts in national assessments of writing. A number of additional correlational analyses and unplanned analyses were performed on this data. First of all, meaningful revision scores were not significantly correlated with the IWA, the holistic scores, or ~position~g frequency within text. However, as predicted, surface changes in Paper 3 were inversely correlated with students’ holistic scores: r = -.45, p < .OOl. Next, scroll~~reposition~g within text was significantly correlated with the holistic scores: I = .49, p < -001. Finally, the holistic scores for Paper 3 and the IWA were significantly correlated: r = .33, p < .OOl. In addressing possible attitude differences between groups toward using the computer as a writing tool, post hoc ANOVAs were performed on the 10 items of the computer questionnaire. ANOVAs for Items 3 and 7 displayed main effects for group: F(1, 148) = 4.07, p < .05; 1;(1, 150) = 4.40, p < .05. The treatment group responded more positively to Question 3 (whether the computer helped them to think about the reader more than when they wrote with pen and paper) (latent: X = 4.78, SD = 2.83; control: X = 3.84, SD = 2.61). Similar results were obtained for Question 7 (whether there were things they could write, think about, or do with the computer that they could not do with pen and paper) (treatment: X = 4.54, SD = 3.02; control: X = 3.51, SD = 3.02). Main effects for group were not significant on the other 8 items of the questionnaire.

SUMMARY OF THE RESULTS AND DISCUSSION

Writing is a complex problem-solv~g process. It is even more complex for young children who lack a monito~ng system that enables them to switch effectively between the generative and evaluative demands of both planning and revising text. The generative-evaluative intervention used in this study was one attempt to influence children’s metacognitive control of their writing plans and goals, thereby attempting to overcome their usually poor content-generation coping strategies,

Adolescent composing within a generative-evaluative computerizedframework

57

The main source of support for the treatment program devised was in the revisionary strategies undertaken by students exposed to the prompts. However, evidence supporting the usefulness of thinking-skill prompts for early adolescent children within either a particular paper or over time (in this case 6 weeks) was minimal. Perhaps students’ content-generation coping strategies are not easy habits to change. Still, in a short-term intervention with college students using this generative-evaluative prompting framework, Reynolds and Bonk (1990a) found treatment-related increases in both process (meaningful revisionary practices) and product (overall writing quality) variables. Additional analyses comparing the prompt-usage and effectiveness of subjects in these two studies was more revealing (Bonk & Reynolds, 1991). In these further explorations it was in fact determined that evaluative types of prompts were used more often and more productively by college students, resulting in increased writing quality. In contrast, early adolescent subjects favored generative heuristics that had less effect on product quality. Reflections on how these findings fit predictions of the generative-evaluative model is discussed in that article. In this study, development was not only examined between groups and over time, but also explored between three grade and two ability levels. As reported briefly earlier, in most measures there were significant main effects that favored the older and higher-ability students. The grade comparisons performed on the IWA, the open-ended question in writing, errant keystrokes, and surface change scores provided consistent findings: Both eighth graders and seventh graders performed better than sixth graders, but there were no differences between the seventh and eighth graders. Conclusions from these grade and ability writing differences are pursued in follow-up investigations of the IWA (e.g., see Bonk et al., 1991). It appears, however, that targeting metacognitive assistance at the sixth-grade level might prove to be the most beneficial. The lack of positive effects of the prompts on students’ writing performance should not immediately discount their utility. First, because this study was longer and spread across more grade levels than prior procedural facilitation interventions, it is conceivable that the generic prompting utilized here may need more fine tuning to address specific task, grade, and ability demands. Second, because overall prompt productivity (i.e., a prompt causing a meaningful change coinciding with the intent of the prompt) was lower than expected (only about 14%) and students only used approximately 16 prompts per paper, improved writing performance failed to occur. Perhaps younger and lower ability children require more system control and mediation of prompt selection to increase exposure and effectiveness. Third, though thinking skill-related dimensional scoring corresponding to the eight prompt categories employed was developed and successfully tested, it too may need further refinement (see Table 2). Fourth, the fact that student performance generally did not decrease in this novel writing environment suggests that students can readily adapt to different types of word processing environments. For example, the layout of the prompts within the bottom two lines of the screen did not appear to inhibit performance. Children did not complain about the loss of two lines of te:ct. In fact, the computer attitudes questionnaire indicated that the prompts helped the treatment group think of potential readers and accomplish more things than they could with pen and paper. Note, however, that prompts were most effective at the sixth-grade level. Finally, as indicated, a more optimistic comparison of prompt effectiveness by grade, prompt category, and composing session productiveness is reviewed in Bonk and Reynolds (1991).

58

Bonk and Reynolds

The testing a more inclusive model of procedural facilitation in writing - a generativesvaluative framework for prompted writing - disclosed changes in students’ revisionary patterns as a result of access to the prompts. Students in the prompted condition were producing fewer surface-level changes when exposed to the prompts, as compared to both their earlier performance and the control group. In addition, the treatment group increased the number of repositions they made within text when prompts were available in Paper 3. Apparently, the prompts caused the treatment subjects to scan their text more frequently than the control subjects. As indicated by a two-way interaction, this was especially true of highability subjects in the treatment condition. Nevertheless, the combination of reduced time spent on surface-level aspects of text and the additional time allocated to the scrolling of one’s text did not result in either more meaningful changes or better quality texts. Maybe the students were pondering over the prompts but did not know how to respond to them. However, if that were true, one would have predicted that the older students would have made more effective use of the prompts than the younger students. A more plausible explanation is that the categorization scheme used to code meaningful changes is in need of restructuring (see Faigley & Witte, 1981, or Reynolds and Bonk, 1990b). Although the treatment group performed below original expectations, progress was made in understanding the connection between writing and thinking, as reflected in the design of the dimensional scoring instrument and the correlations among the instruments used to measure metacognition, internalization, and writing performance. Two of the strongest correlations were found between the IWA and the holistic scores, and between the open-ended question in writing and the holistic scores. Thus, it appears that an understanding of one’s writing strategies has an impact on the quality of text produced. Students who have an understanding of how to plan, evaluate, regulate, and conditionally apply writing strategies appear to produce higher quality texts. Though this is intuitively simple, the impact of the IWA may be significant for diagnosticians attempting to measure the success of their metacognitive interventions and English teachers searching for an easily administered diagnostic tool. As noted earlier, additional work is warranted in replicating these preliminary results with the IWA. Afterwards, the IWA should be extended across grade levels and modified to eliminate possible context-related bias in strategic scoring (Bonk et al., 1991). There are a number of treatment variations from the current program that might be utilized to further explore the metacognitive and performance effects of prompts within children’s writing. For instance, students in future interventions could receive greater initial modeling of the prompt categories. In addition, student involvement in the creation of prompts and prompt categories along with the use of the prompts for a semester or academic year also could eliminate some of the problems encountered here.5 Yet, before commencing on either of these directions, researchers must ask themselves if the extra time spent within an individual classroom or school would be fruitful. Is their enough merit in the ideas behind procedural facilitation, keystroke analysis, and internalization of prompts to warrant the continuation of this research? Keystroke analyses enabled the experimenters to document when and where prompts were most effective. For instance, sixth graders and high-ability students were more productive in using the prompts than older and lower ability students, respectively. Secondary comparisons with the parallel college students’ study (Reynolds & Bonk, 1990a) point toward the usefulness of procedural facilitation

Adolescent composing within a generative-evaluative computerizedframework

59

and keystroke mapping in understanding patterns within prompt selection, prompt productivity rates, and age differences in composing strategies (Bonk & Reynolds, 1991). Additionally, keystroke replay contributes to understanding the relationship between revisionary practices and writing quality, the number and type of revisions within and between drafts, and individual differences in text production (Reynolds and Bonk, 1990b). As a result, we advocate continued use of keystroke replay and other adjunct writing aids to analyze the generative-evaluative processes of writing. Additional investigations might be undertaken to increase our understanding of the effectiveness of prompts within writing (i.e., why prompts may be more useful at certain grades, ability levels, and moments in the composing process). One possible study might focus on the usefulness of prompts at the sixth-grade level using multiple treatment groups (e.g., one group receiving generic generative and evaluative prompts used in the present study, a second group receiving topic-specific prompts, and a third treatment group creating and using their own prompt categories and individual prompts). The present study also illustrates that the computer tools (and other media) cannot be separated from instruction. Perhaps more positive effects would have resulted from the linkage of the computer-prompt intervention to actual language arts instruction. Effective writing instruction suggests that introducing the objectives and principles of the prompts in the classroom prior to encountering them on the computer might facilitate their usage (Hillocks, 1986). Studies that integrate new prompting media with appropriate instruction should increase the possibility of strategy internalization. Moreover, studies that increase our understanding of how to facilitate thought processes involved in superior writing performance are critical to the evolution of writing research. At the present time, however, few studies meet that goal. Our study was one attempt to expand the envelope of what is known about the connection between writing and reasoning. The additional studies proposed above (e.g., those involving greater integration with classroom instruction, modeling, student involvement, and peer feedback) should further this understanding and also the utility of models of the generative-evaluative processes in composing. Acknowledgments - The research here was supported in part by dissertation research grants obtained by the first author from the American Psychological Association (APA) and the Wisconsin Alumni Research Foundation (WARP). The authors wish to thank Gary Davis, Steve Yussen, Richard Lehrer, Joan Littlefield, Mary Gomez, Jim Middleton, Todd Franke, Robert Gilpatrick, Nancy Holty, Cheryl Hofstad, Cathy Salapa, and Barb Seeboth for their support and guidance with this project, and the many other students and teachers who made this study possible.

NOTES

1. Of course, the authors are indebted to the cognitive process model of writing (Flower & Hayes, 198 1) and corresponding knowledge telling/knowledge transformation research (Bereiter & Scardamalia, 1987). 2. These students represented the entire school enrollment at these grade levels. 3. The package used was WordPerfect 4.2 from WordPerfect Corporation. 4. If it helps, think of computer-controlled keystroke replay as reminiscent of watching a player piano in an old Western movie. 5. Scott Paris (personal communication, 1989) concluded that widespread implementation of pragmatic treatments like this one would require one year or more for significant variation to be detectable across classes.

60

Bonk and Reynolds

REFERENCES

Applebee. A. N. (1982). Writing and learning in school settings. In M. Nystrand (Ed.), What writer’s know: The language, process, and structure of written discourse (pp. 365-381). New York: Academic Press. Applebee. A. N. (1984). Writing and reasoning. Review of Educational Research, 54.577-596. Bar&t, L. S., & Kroll, B. M. (1978). Some implications of cognitive-developmental psychology for research in composing. In C. R. Cooper & C. R. Odell (Eds.), Research on composing (pp. 49-57). Urbana, IL: National Council of Teachers of English. Bartlett, E. J. (1982). Leaming to revise: Some component processes. In M. Nystrand (Ed.), What writer’s know: The language, process, and structure of written discourse. New York: Academic Press. Bereiter, C. (1980). Development in writing. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing. Hillsdale, NJ: Erlbaum. Bemiter, C., & Scardamalia, M. (1982). From conversation to composition: The role of instruction in a developmental process. In R. Glaser (Ed.), Advances in instrumental psychology (Vol. 1, pp. l-64). Hillsdale, NJ: Erlbaum. Beteiter, C., & Scardamaha, M. (1985). Cognitive coping strategies and the problem of “inert knowledge.” In J. W. Segal. S. F. Chipman, & R. Glaser @is.), Thinking and learning skills: Vol. 1. Research ana’open questions (pp. 65-80). Hillsdale, NJ: Erlbaum. Bereiter, C., & Scardamalia, M. (1987). The psychology of written communication. Hillsdale, NJ: Erlbaum. Bonk. C. J. (1989). The effects of generative and evaluative computerized prompting strategies on the development of children’s writing awareness and performance. Dissertation Abstracts International, 51,789A. (University Microfilms No. 89-23,313) Bonk, C. J. (1990). A synthesis of social-cognition and writing research. Written Communication, 7, 136-163. Bonk, C. J., Middleton, J. A., Reynolds, T. R., & Stead, F. L. (1991). The Index of Writing Awareness: A tool for measuring early adolescent metacognition in writing. Paper presented at the American Educational Research Association annual meeting, Chicago. Bonk, C. J., & Reynolds, T. H. (1991). Repurposing the word processor for reprocessing behaviors: Prompting questionsprompting questions. Manuscript submitted for publication. Bridwell, L. S., Nancarrow, P R., & Ross, D. (1984). The writing process and the writing machine: Current research on word processors relevant to the teaching of composition. In A. Beach & L. S. Bridwell (Eds.), New directions in composition research (pp. 381-398). New York Guilford. Bridwell, L., Sire, G., & Brooke, R. (1985). Revising and computing: Case studies of student writers. In S. W. Freedman (Ed.), The acquisition of written language (pp. 172-194). Norwood, NJ: Ablex. Caccamise, D. J. (1987). Idea generation in writing. In A. Matsuhashi (Ed.), Writing in real time: Mo&lling production processes. Norwood, NJ: Ablex. Carey, L. (1988). Pushing planning to the limit: Using computer based prompts to elicit writers’ plans. Paper presented at the American Educational Research Association annual meeting, New Orleans, LA. Collins, A., & Genmer, D. (1980). A framework for a cognitive theory of writing. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 51-72). Hillsdale, NJ: Erlbaum. Comeaux, M.. & Lehmr, R. (1986). The mind of a writer: A cognitive model of computer-assisted composition. Paper presented at the American Educational Association annual meeting, San Francisco, CA. Daiute, C. (1985). Do writers talk to themselves? In S. W. Freedman (Ed.), The acquisition of written language: Response and revision. Norwood, NJ: Ablex. Daiute, C. A. (1986). Physical and cognitive factors in revising: Insights from studies in computers. Research in the Teaching of English, 20(2), 141-159. Davis, G. A. (1986). Creativity isforever (2nd ed.). Dubuque, IA: Kendall/Hunt. Educational Testing Service. (1984). Holistic scoring guide. Princeton, NJ: Author. Englert, C. S., Raphael, T. E., Fear, K. L., & Anderson, L. M. (1988). Students’ metacognitive knowledge about how to write information texts. Learning Disability Quarterly, 11, 18-46. Faigley, L., & Witte, S. (1981). Analyzing revision. College Composition and Communication, 33, 400-414. Fitzgerald, J. (1987). Research on revision in writing. Review of Educational Research, 57,481-506.

Adolescent composing within a generative-evaluative

computerized framework

61

Flower, L. (1987). The role of task representation in reading to write (Tech. Rep. No. 6). Berkeley, CA & Pittsburgh, PA: University of California & Carnegie Mellon University, Center for the Study of Writing. Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and Communication, 32,365-387. Flower, L., Hayes, J. R., Carey, L., S&river, K., & Saatman, J. (1986). Detection, diagnosis, and the strategies of revision. College Composition and Communication, 37.16-55. Freedman, S. W. (1987). Response to student writing. Urbana, IL: National Council of Teachers of English. Glynn, S. M., Oaks, D. R.. Mattocks, L. F., & B&ton, B. K. (1989). Computer environments for managing writers’ thinking processes. In B. K. Britton & S. M. Glynn (Eds.), Computer writing environments: Theory, research, and design (pp. 1-16). Hillsdale, NJ: Erlbaum. Haas, C. (1988). How word processors affects planning in writing: The impact of technology. Paper presented at the American Educational Research Association annual meeting, New Orleans, LA. Hawisher, G. E., & Fortune, R. (1988). Research into word processing and the basic writer. Paper presented at the American Educational Research Association annual meeting, New Orleans, LA. Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. American Psychologist, 41, 1106-1113. Higgins, L., Flower, L., & Petraglia, J. (1990). Planning text together: The role of critical reflections in student collaboration. Paper presented at the American Educational Research Association annual meeting, Boston, MA. Hillocks, G. (1986). Research on written composition: New directions for teaching. Urbana, IL: National Conference of Research in English & ERIC Clearinghouse on Reading and Communication Skills. Jacobs, J. E., & Paris, S. G. (1987). Children’s metacognition about reading: Issues and definition, measurement, and instruction. Educational Psychologist, 22,255278. Kellogg, R. T. (1989). Idea processors: Computer aids for planning and composing text. In B. K. Britton & S. M. Gly~ (Eds.), Computer writing environments: Theory, research, and design (pp. 57-92). Hillsdale, NJ: Erlbaum. Kozma, R. B. (in press). The impact of computer-based tools and embedded prompts on writing processes and products of novice and advanced college writers. Cognition and Instruction, Martlew, M. (1983). Problems and difficulties: Cognitive and communicative aspects of writing development. In M. Martlew (Ed.), The psychology of written language: Developmental and educational perspectives. New York: Wiley. McCutchen, D. (1988). “Functional automaticity” in children’s writing: A problem in metacognitive control. Written Communication, 5,306-324. Montague, M. (1990). Computers, cognition, and writing instruction. Albany: State University of New York Press. Purves, A. C. (1985). Framework for scoring: GREIToej7. Unpublished manuscript, University of Illinois at Urbana-Champaign, Curriculum Laboratory. Reynolds, T. H., & Bank, C. J. (1990a). Facilitating college writer’s revising processes. Paper presented at the American Educational Research Association annual meeting, Boston, MA. Reynolds, T. H., & Bonk, C. J. (1990b). Windows on writing: The usefiilness of keystroke mapping to monitor writing progress. Paper presented at the 6th Computers and Writing Conference, Austin, TX. Ruth, L., & Murphy, S. (1988). Designing writing tasks for the assignment of writing. Norwood. NJ: Ablex. Salomon, G. (1988). AI in reverse: Computer tools that turn cognitive. Journal of Educational Computing Research, 4,123-139. Scardamalia, M., & Bereiter, C. (1982). Assimilative processes in composition planning. Educational Psychologist, 17, 165-171. Scardamalia, M., & Bereiter, C. (1985). Fostering the development of self-regulation in children’s knowledge processing. In J. W. Segal, S. F. Chipman, 8z R. Glaser (Eds.), Thinking and learning skills: Vol. I. Research and open questions (pp. 563-577). Hillsdale, NJ: Erlbaum. Scardamalia, M., 8z Bemiter, C. (1986). Research on written composition. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 778-803). New York: Macmillan. Scardamalia, M., Bemiter, C., & Steinbach, R. (1984). Teachability of reflective processes in written composition. Cognitive Science, 8, 173-190. Smith, J. B., & Lansman, M. (1989). A cognitive basis for a computer writing environment. In B. K. Britton & S. M. Gly~ (Eds.), Computer writing environments: Theory, research, and design. Hillsdale, NJ: Erlbaum.

Bonk and Reynolds

62

Talbot, J. (1986). The assessment of critical thinking in history/social science through writing. Social St&es Review, 25(2), 33-41.

Torrance, E. R (1974). Norms and technical manual: Torrance Tests of Creative Thinking (rev. ed.). Bensenville. IL: Scholastic Testing Service. Vygotsky, L. (1986). Thought and language (rev. ed.). Cambridge, MA: MIT Press. Witte, S. R (1985). Revising, composition theory, and research design. In S. Freedman (Ed.), The acquisition of written language: Response and revision (pp. 250-284). Not-wood, NJ: Ablex. Witte, S.. & Faigley. L. (1981). Coherence, cohesion, and writing quality. College Composition and Communication, 32,189-204.

Woodruff, E., Bereiter, C., & Scardamalia, M. (1981). Gn the road to computer assisted composition. Journal of Educational Technology Systems, 10.133-148. Zellermayer, M., Salomon, G., Globerson, T., & Givon. H. (1991). Enhancing writing-related metacognitions through a computerized writing partner. American Educational Research Journal, 28,373-391.