Integrating written metacognitive awareness guidance as a ‘psychological tool’ to improve student performance

Learning and Instruction 13 (2003) 633–651 www.elsevier.com/locate/learninstruc

Integrating written metacognitive awareness guidance as a ‘psychological tool’ to improve student performance E. Guterman ∗ National Institute of Child Health and Human Development, One Rockledge Center, Suite 8048, 6705 Rockledge Drive, Bethesda, MD 20892-7970, USA

Abstract A study was designed to test the effect of using MetaCognitive Awareness Guidance (MCAG) as a vehicle for activating and engaging learners’ existing prior knowledge before they begin a reading assessment task. A total of 300 pupils (aged 9–10) studying in ten 4th grade classes in four different schools in Israel participated in the research. The major assumption of the study was confirmed: increasing learners’ metacognitive awareness by means of well-planned guidance, built on prior knowledge, facilitates their learning and improves outcomes on a reading assessment task. This paper focuses on the linkage between three components of learner performance, what we refer to as the ‘Assessment Triangle’: (1) achievement on reading assessment tasks; (2) performance on metacognitive awareness guidance; and (3) awareness of metacognitive reading strategies. The implications and the function of this revealed linkage will be discussed. Published by Elsevier Ltd.

1. Introduction This paper describes a study designed to test the effect of using MetaCognitive Awareness Guidance (MCAG) as a vehicle for activating and engaging learners’ existing prior knowledge before they begin a reading assessment task (Guterman, 2000—Metacognitive awareness and performance on assessment. Tasks in reading. Unpublished doctoral thesis, University of Liverpool, UK). A central belief underlying the study is that the assessment of reading ability should be based on a model ∗

Tel.: +1-301-496-0420; fax: +1-301-480-5775. E-mail address: [email protected] (E. Guterman).

0959-4752/$ - see front matter Published by Elsevier Ltd. doi:10.1016/S0959-4752(02)00070-1

634

E. Guterman / Learning and Instruction 13 (2003) 633–651

of cognition and learning which itself rests on the best available understanding of the learning process. This model should reflect current theory regarding how students represent knowledge and develop competence in a domain, and should serve as a unifying element in linking curriculum, instruction, and assessment. This cohesive function is crucial because educational assessment does not exist in isolation, but must be aligned with curriculum and instruction if it is to support students’ learning.

2. Theoretical framework The written MetaCognitive Awareness Guidance (MCAG) described in this study was a set of open questions given students before they were to perform a reading assessment task, which aimed to raise their metacognitive awareness, and prepare them for dealing with the task. It affords learners the opportunity to engage in higher order operations: to test their knowledge, discover new links, anticipate, raise questions, suggest possible answers, etc. As such, the MCAG functions within the zone of proximal development, and addresses the learner’s potential level of development. It was constructed on the basis of three contemporary theories of learning and knowledge: schema theory, metacognitive awareness theory and the Vygotskian ‘zone of proximal development’. 2.1. Prior knowledge and schema theory Prior knowledge is defined in the research as the various kinds of information contained in learners’ long-term memory, that they need in order to construct meaning from written text: linguistic knowledge, grammar knowledge, skill knowledge, strategy knowledge, textual knowledge, contextual knowledge, overall prior knowledge, specific prior knowledge, and world knowledge. Prior knowledge provides a framework that allows readers to organize text information more efficiently and effectively and to integrate the new information into the old, which facilitates retention. It allows readers to make inferences about what happened or is likely to happen in the text, thus helping learners to fill in gaps in the material. Prior knowledge also helps readers to elaborate upon the material. Elaboration, a cognitive activity that involves speculation, judgment, and evaluation, is a powerful aspect of reasoning with print. Prior knowledge can best be understood through schema theory, based on the work of Anderson and colleagues (Anderson, Reynolds, Shallert, & Goetz, 1977; Anderson, 1983; Anderson & Pearson, 1984), Anderson (1980); Rumelhart (1980) and Bransford and Johnson (1972), among others. Schema theory attempts to explain how knowledge is represented in the mind, and how that representation facilitates the use of knowledge. In accordance with schema theory, all knowledge is packed into units referred to as schemata. Each of these is a ‘packet of knowledge’ which includes (1) what one knows about a concept, subject or issue; (2) how other, different kinds of information are related to that concept, subject or issue; and (3) how this knowledge is to be used (Rumelhart, 1980).

E. Guterman / Learning and Instruction 13 (2003) 633–651

635

Schemata have been called the ‘building blocks of cognition’ (Rumelhart, 1980) and a cognitive map of the world (Neisser, 1976) because they represent the abstract structure of knowledge and the elaborate networks of concepts, skills, and procedures which we use to make sense of new stimuli, events, and situations. According to schema theory, comprehension is the use of prior knowledge to create new knowledge. Without prior knowledge (which must be activated to be useful), written material would be meaningless. The more knowledge a reader can bring to a text, the more likely it is that the written material will be understood (Rumelhart, 1980; Wilson & Anderson, 1986). These and other researchers do not simply argue that the activation of appropriate prior knowledge is useful; they assert that it is fundamental to comprehension and performance. 2.2. Metacognitive awareness The design of the written Metacognitive Awareness Guidance (MCAG) in this study was based on current theories of human cognition and learning, dealing with metacognitive skills, and their functions and roles in understanding, learning, constructing new meanings, creating new knowledge and solving problems (Flavell, 1976; Anderson, 1980; Bereiter & Scardamalia, 1987; Brown, Campione, Reeves, & Ferrara, 1990; Bransford, Brown, & Cocking, 1999; Pellegrino, Chudowsky, & Glaser, 2002) and on the relevant research that put these principles to work (Brown, Smiley, & Lawton, 1978; Brown, Campione, & Barclay, 1979; Anderson & Armbruster, 1980; Bransford, Stein, Shelton, & Owings, 1980; Spiro & Tierre, 1980; Bransford, Stein, Arbitman-Smith, & Vye, 1985). The terms metacognition, meta-comprehension, meta-linguistics and metacognitive awareness are crucial to the issue of activating schemata. Basically, these terms assume that the reader is: 앫 aware of the various factors involved in the learning situation—the characteristics of the text, the requirements of the task, applicable strategies and the learner’s own abilities and deficiencies 앫 aware of different kinds of prior knowledge 앫 aware that different kinds of prior knowledge can and need to be used in the attempt to learn and construct meaning 앫 aware that using this prior knowledge and background will affect the ability to learn from reading and will affect learning performance 앫 able to use this prior knowledge consciously One of the main conclusions of this is that when we talk about metacognitive awareness, we are actually referring to knowledge and knowing, to being aware of knowledge in order to know. A reader’s knowledge and awareness of this knowledge will determine the level of his or her understanding, performance and achievement. Within the framework of the study described here, we examined the extent to which providing readers with metacognitive awareness guidance (MCAG) prior to reading a text, would improve their comprehension of the text. The fundamental

636

E. Guterman / Learning and Instruction 13 (2003) 633–651

purpose of the MCAG was to raise learners’ metacognitive awareness of their prior knowledge: both specific prior knowledge (text-specific and topic-specific) and overall prior knowledge, before they engage in processing their reading tasks. However, the main theoretical rationale underlying the combination of written metacognitive awareness guidance with a reading assessment task was Vygotsky’s notion of a zone of proximal development—ZPD (Vygotsky, 1978; Moll, 1990). 2.3. Vygotsky’s zone of proximal development Vygotsky (1978) argued that one cannot understand the child’s level of development unless one considers two aspects: the actual development level and the potential development level. He argued that measuring the level of potential development is just as crucial, if not more so, as measuring the level of actual development because the potential level of development may vary independently of the actual level. This concept served as a major guideline for developing and using Metacognitive Awareness Guidance (MCAG): performance must be assisted, and assisting performance by combining a ‘psychology tool’ and a stimulus with an assessment task will make a difference in the learner’s performance and outcomes (Brown & French, 1979; Brown & Ferrara, 1985; Campione, Brown, Reeve, Ferrara, & Palincsar, 1991). In Vygotsky’s words, a ‘psychological tool’ serves “as a conductor of humans’ influence on the object of their activity. It is directed towards the external world; it must stimulate some changes in the object; it is the means of humans’ external activity directed towards the subjugation of nature” (Vygotsky 1960, in Wertsch, 1985, p. 78). The stimulus or the scaffold given to learners becomes, in time and with social support, part of the learner’s repertoire of understanding. Implicitly, what we are first able to do with others, we are eventually able to do by ourselves. In this study, the stimulus—the written MCAG—provides the assistance of others in situations where learners are encouraged to perform activities more mindfully and meaningfully, before attempting to process assessment tasks.

3. The assessment triangle Using the MCAG together with reading assessment tasks effects three components of learner performance: (1) achievement on reading assessment tasks; (2) performance on metacognitive awareness guidance (MCAG); and (3) awareness of metacognitive reading strategies. These are the three segments of the students’ Assessment Triangle. This part of the paper will analyze the linkage between the these three components and its role in learners’ cognitive performance. 3.1. The first component of the ‘assessment triangle’: the reading assessment task The study made use of actual reading assessment tasks that had been developed, tested, and are currently being implemented by the Israeli Ministry of Education.

E. Guterman / Learning and Instruction 13 (2003) 633–651

637

Three reading assessment tasks were chosen from the “Kits of Assessment Tasks” (KATs) in Reading (Director General’s special circular in Hebrew, Ministry of Education, 1996). Each KAT includes up to 40 tasks in one subject for one grade level, accompanied by a rationale, a teacher’s guide containing a brief explanation of the nature of the tasks and a scoring rubric. The KAT for “Reading Comprehension and Writing for Grade 4” includes a literary text (16 tasks), an informative text (seven tasks), media excerpts (seven tasks), a “how to” type text—guidelines for preparing something (three tasks), visual texts (two tasks), and ancient Judaic texts (two tasks). According to Lewy (1997), tasks in the collection are authentic, challenging and attainable (pp. 25–30). They are authentic in that they aim to model real learning activities taken from real contexts (within the students’ immediate environment in space and time), which come from authentic sources and are presented in their original form. The assessment tasks in the collection reflect the ability of the student to cope with authentic unseen texts on various levels of difficulty and complexity among the existing inventory of texts in the students’ cultural environment (Inbar, 1995, p. 2). A challenging task is defined as a task that provides a cognitive, social or personal challenge, preferably using open-ended questions rather than multiple-choice items. In open-ended questions, students perform complex tasks: they formulate answers, carry out operations and describe them, express ideas in their own words, collect data, integrate data drawn from various sources, justify statements, interpret differences in approaches, identify advantages and disadvantages of a given action—in short, students use their minds. Attainable tasks are those that have been tried out on a representative sample of the target population. Results have shown empirically the feasibility of attaining the prescribed learning objectives. Thus, it is unlikely that student failure is the result of setting an inappropriate target for the test population, lack of clarity, or other aspects related to the task itself. For this study, the three reading assessment tasks chosen (which represent the major genres of reading in the KATs) were “How Paper is Made” (task 1); The End (task 2); and We Dream (task 3). Task 3 is shown in Fig. 1. 3.2. The second component of the ‘assessment triangle’: MetaCognitive Awareness Guidance (MCAG) The fundamental purpose of the MCAG was to raise learners’ metacognitive awareness of their prior knowledge: both specific prior knowledge (text-specific and topic-specific) and overall prior knowledge, and to allow them to reflect on their existing prior knowledge before they begin their reading assessment tasks. More specifically, it aimed to help the students: 앫 become aware of what they already know about the topic of a text before answering comprehension questions. This was accomplished by providing readers with information about the text, for example, The reading passage that you are about to read is from…and it is called...; You are about to read two different passages… written in... by…; The writer was born…; She wrote and translated....

638

E. Guterman / Learning and Instruction 13 (2003) 633–651

Fig. 1.

Assessment task 3—We dream.

앫 concentrate on and invest mental effort in constructing meaning rather than identifying correct meaning, by focusing on what they already know and understand. This was accomplished by giving tasks such as: Write a short paragraph to fit the title…using the following words…; Write your own thoughts and ideas about...; Complete the missing words and explain why…; Give a reason for your answers…. 앫 create an ‘advanced organizer’ for the task by building bridges and making connections between existing knowledge and the ideas which are communicated in the text. This was done by asking the learners: What do you think...?; What could be a problem...?; What might be a solution...?; In your opinion, what…?; Write possible answers to the questions you have asked.

E. Guterman / Learning and Instruction 13 (2003) 633–651

639

앫 make an active effort to construct meaning by predicting what the text may be about, by leading the learners to make judgments about how new information relates to what they already know, so that they can fit new pieces into the partiallyassembled puzzle which already exists in their minds. For this purpose, readers were asked to write, What do you think…?; Have you any ideas about…?. 앫 focus on their existing knowledge by asking them to discover for themselves what they already know about a topic, and direct this knowledge to their working memory. For example, Since I know … and …, I can presume that…; Now I know…; I am sure of … and …, and this leads me to think that…. 앫 activate their existing knowledge by anticipating, raising questions, suggesting possible answers, and extending thinking about the task/topic/issue before going to work on it. To achieve this, the MCAG employs direct explicit self-talk. Learners are asked to stop and reflect on what they did, why they did it, and how to use what they did. This breaks down their spontaneous tendency to start working—to answer the question immediately. Through the use of self-talk, their processing of the reading tasks becomes less impulsive and more mindful. After answering the written MCAG questions, learners are instructed: Now, before beginning the reading comprehension task, please say out loud: ‘Now I know more about...; What I know about ... will help me to understand the text and to answer the questions’. The use of self-directing talk is a major theme in Vygotskian development theory. At one point, children begin to use language not only to communicate but to guide, plan, and monitor their activity. In Vygotsky’s own words: The specifically human capacity for language enables children to provide for auxiliary tools in the solution of difficult tasks, to overcome impulsive action, to plan a solution to a problem prior to its execution, and to master their own behavior (Vygotsky 1978, p. 28). The written MCAG for Task 3 is shown in Fig. 2. 3.3. The third component of the ‘assessment triangle’: The Metacognitive Strategy Index questionnaire (MSI) The Metacognitive Strategy Index questionnaire (MSI) was based on a questionnaire designed by Schmitt (1990) to measure children’s awareness of metacognitive reading strategies. The questionnaire was originally developed to measure the strategic awareness of students who participated in a meta-comprehension training study (Schmitt, 1988) and adaptations of the index have been used in several additional studies (e.g., Baumann & Schmitt, 1987; Lonberger, 1988). The strategies assessed by the MSI are consistent with those taught in several meta-comprehension instructional studies (e.g., Palincsar & Brown, 1984; Braun, Rennie, & Labercane, 1986; Riscko & Feldman, 1986). Each of the 25 questions had four distracters, a total of 100 statements, and the questions related to three time periods: before, during and after reading. Students were asked to read the four statements and decide which would help them the most to understand the story. The instructions clarify that there are no right answers. It is just what you think would

640

E. Guterman / Learning and Instruction 13 (2003) 633–651

Fig. 2.

Metacognitive Awareness Guidance—Task 3.

E. Guterman / Learning and Instruction 13 (2003) 633–651

641

help the most. Schmitt classified the questions according to six metacognitive awareness strategies: predicting and verifying; previewing; purpose setting; self-questioning; drawing on background knowledge; summarizing and applying fix-up strategies (Schmitt, 1990, p. 455). The MSI has been shown to be a reliable measure of meta-comprehension strategy awareness. Lonberger (1988) reported an MSI internal consistency value of 0.87 using the Kuder-Richardson Formula 20 (6). In addition, Schmitt (1988) found in an experimental study that students who received training in meta-comprehension strategies scored significantly higher on the MSI than students in a non-instructed group. She also found statistically significant correlations between the MSI and two comprehension measures, an error detection task (r = 0.50, p ⬍ 0.001) and a cloze task (r = 0.49, p ⬍ 0.001) (1988, p. 455). These data provide further evidence of the relationship between performance on the MSI and tasks commonly used to measure students’ meta-comprehension ability. Though activation of prior knowledge is included in all of Schmitt’s categories, a more general category was defined for this research: BCD—building, creating and discovering relevant prior knowledge. The data obtained from the MSI was analyzed on four levels. The first was the overall level of metacognitive awareness. The second was the level of metacognitive awareness for each of the three parts—before, during and after reading. The third was the level of metacognitive awareness for each of the six categories. The last was the level of metacognitive awareness for BCD. The reliability of the 17 items of the Hebrew version of the MSI used in this research was found to be 0.68.

4. Method 4.1. Participants A total of 300 pupils (aged 9–10) studying in ten 4th grade classes in four different schools in Israel participated in the research. Two of the schools were located in an upper middle class neighborhood and the other two schools were located in a lower middle class neighborhood. 4.2. Procedure The study utilized three modalities: A control group, which received no intervention, a placebo group that received content instruction (CI) guidance, and the intervention group, that was given written Metacognitive Awareness Guidance (MCAG). The first version was designed for the control group (N = 102). The students were asked to read the text and answer the questions that followed. This was the procedure set forth by the developers of the KATs (Director General’s special circular in Hebrew, Ministry of Education, 1996). The second version was especially designed to reduce a possible halo effect, i.e., performance affected by being part of a special procedure. It was given to students

642

E. Guterman / Learning and Instruction 13 (2003) 633–651

in a placebo group (N = 85) who were asked to read Content Instructions (CI) and the text only, and when they finished reading, to raise their hands to signal that they were ready to proceed to the reading assessment tasks. The CI consisted of instructions that focused on content and procedure, i.e., “Read the passage... carefully. When you finish reading it, you will be asked questions on what you read. Pay attention. Before you begin to answer the question, be sure that you understand the passage.... After you answer all the questions, go back and check your answers.” The third version, the fully developed target tool—the written MetaCognitive Awareness Guidance (MCAG)—was given to the treatment group (N = 113). It included written metacognitive awareness guidance for each of the three reading assessment tasks. The MCAG was given to students before performing the reading assessment tasks. After answering the metacognitive awareness questions, they were asked to say out loud, “Now I know a lot about.... What I know about... will help me to understand the passage. Now it will be easier for me to study the passage,” and only then to signal that they were ready to process the assessment task. The MSI was administered to all three groups 14 days after the three assessment tasks were performed. The achievements of the learners in the treatment group were compared to those of learners in the control group who performed the same tasks without any support, and to the achievements of learners in the placebo group who received content instruction and read the text of the tasks before processing them. Learners’ level of achievement was also examined in relation to other research variables: gender, school, and level of awareness of metacognitive reading strategies.

5. Findings 5.1. Pupils’ achievement on the three reading assessment tasks Table 1 presents pupils’ achievements on the three reading assessment tasks. Oneway ANOVA performed on the scores of each of the three assessment tasks and on the achievement for all tasks, reveals a significant difference among the research groups in level of achievement for each of the tasks. The subjects of the treatment group attained a significantly higher score than the control group (the non-treatment group) and the CI group that received the placebo treatment on each of the three assessment tasks and on all tasks together. The Sidak t-test for multiple post hoc comparisons was used to determine which of the mean scores differed significantly from the others. As Table 1 shows, Group 3, the treatment group, was found to be significantly different across the three reading assessment tasks - ‘Paper’: 3 ⬎ 2, 3 ⬎ 1; ‘End’: 3 ⬎ 2, 3 ⬎ 1; ‘Dream’: 3 ⬎ 1, 2 ⬎ 1; and ‘All tasks’: 3,2 ⬎ 1, 3 ⬎ 2, 2 ⬎ 1. The last column of Table 1 shows the effect size (ES) of the intervention (MCAG) on each of the three tasks, and for all tasks together, describing how well the average student who received the intervention (MCAG) preformed relative to the average students who did not received the intervention. The positive large effect size on reading assessment 1- Paper (ES = 0.67), reading assessment 2 - End (ES = 0.72), reading assessment 3 - Dream (ES =

b

a

10.51 1.66 1.43 1.64 1.59

2.66 0.53 0.68 0.58 0.47

Control group 1 (N = 102) Mean SD

Research Group

10.45 1.76 1.58 1.97 1.75

2.30 0.55 0.70 0.48 0.47

Content instruction group 2 (N = 85) Mean SD 9.92 2.02 1.92 1.87 1.96

2.71 0.55 0.69 0.46 0.42

Metacognitive treatment group 3 (N = 113) Mean SD 1.88 38.53 34.41 26.42 17.44

F2,297

Post-hoc comparisons were performed using the Sidak t-test for multiple comparisons Metacognitive Awareness Guidance

Reading Paper End Dream All tasks

Task

Table 1 Achievement on reading and assessment tasks

0.1890 0.0001 0.0001 0.0001 0.0001

p- value

3⬎2, 3⬎1 3⬎2, 3⬎1 3⬎1, 2⬎1 3⬎2⬎1, 3⬎1

Significant post-hoc analysesa

0.67 0.71 0.44 0.83

MCAGb group vs Control group

Effect size (ES)

E. Guterman / Learning and Instruction 13 (2003) 633–651 643

644

E. Guterman / Learning and Instruction 13 (2003) 633–651

0.44) and All tasks (ES = 0.83) indicates that the intervention made a powerful difference in the outcome measured—students achievement on the reading assessment tasks. Through analysis of covariance (ANCOVA) differences between achievements were shown on the reading assessment tasks after the effect of the learners’ prior level of reading ability was removed (Reading). The analysis yielded significant differences among the three groups after the removal of the covariant effect. Two-way ANOVA was performed on the scores for each assessment task and for all three together by treatment and gender and by treatment and school. The results showed no significant interaction between these variables and the treatment. The only statistically significant effect on pupil achievement was the treatment. 5.2. Pupils’ achievement on the MCAG questions Table 2 summarizes the students’ responses to the MCAG questions for the text “We Dream”. Analysis of the students’ answers to MCAG questions were in general very disappointing. Only a small number of pupils could provide satisfactory answers. The majority gave superficial responses to the questions; there seems to be a fundamental gap between having knowledge, on one hand, and actively using it— being able to draw appropriate generalizations and to extrapolate from them—on the other. For example, in question 3, “Do you think it is from the encyclopedia?”, 38%

Table 2 Frequency and percentage of subjects’ responses on MCAG-3 (N=113) No.

Question

Range of scores

1.

Write five questions that you think will be answered in the passage ‘We Dream’.

2.

3.

4.

Frequency

Percentage

0 1 2 3

15 56 42

13.4 49.1 37.5

Of the five questions you have written, select two, and write a possible answer to each question you chose.

0 1 2 3

1 36 69 7

0.9 32.1 60.7 6.3

If you think the sentences below are from the encyclopedia, circle yes or … no.

0 1 2 3

25 45 43

22.3 39.3 38.4

Explain why.

0 1 2 3

66 35 12

58.9 30.4 10.7

E. Guterman / Learning and Instruction 13 (2003) 633–651

645

were able to answer correctly regarding all seven sentences, showing a high level of knowledge, but when asked to explain why (question 4), only 10% did so adequately. 5.3. Pupils’ results on the MSI questionnaire Table 3 shows the results of the analysis of the MSI questionnaire administered 14 days after the completion of the third reading assessment session. One-way ANOVA performed on the total score of the MSI showed significant differences among the three groups in their awareness of metacognitive reading strategies (MSI-T). Regarding the level of awareness before, during and after reading, significant differences were found among groups only ‘before reading’ (MSI-B). The results showed no significant differences among the groups in their awareness of strategies in the during and after portions of the questionnaire. Since the written metacognitive awareness guidance given to Group 3, the metacognitive treatment group, promoted and fostered the metacognitive reading strategy before reading, the linkage between the MCAG and a higher level of awareness is revealed. The link between the metacognitive treatment and a higher level of awareness of reading strategies is strengthened by the findings regarding the six metacognitive strategies as measured by the MSI, five of which show significant differences among the groups (MSI-1 - MSI-5). Group 3 demonstrated the highest level of awareness of the metacognitive reading strategies for predicting and verifying (MSI-1), purpose setting (MSI-3), previewing (MSI-2), self-questioning (MSI-4) and drawing on background knowledge (MSI-5). Four of these strategies, which revealed significant differences between groups, were promoted in the written metacognitive awareness guidance. Regarding the BCD strategies, building, creating, and discovering prior knowledge, which is one of the principles of the metacognitive awareness guidance activity, the group that received metacognitive awareness guidance performed significantly better on this category than the other groups (MSI-BCD). The Sidak t-test showed that the treatment group scored significantly higher than the control and placebo groups, and demonstrated a significantly higher level of awareness of metacognitive reading strategies in each of the above categories. These findings confirm the connection between the MCAG and awareness of metacognitive reading strategies. The MCA treatment group demonstrated a higher level of awareness 14 days after the last reading assessment session. Additional support for this connection is seen in the findings regarding ‘summarizing and applying fixup strategies’. This metacognitive reading strategy was not mentioned in the MCAG either directly or indirectly. It is not surprising, therefore, that no significant difference was found in this category (MSI-6). The connection between the written MCAG and a high level of awareness of metacognitive reading strategies is empowered by the value of the effect size (ES = 0.92). An extremely high and significant ES value of the MCAG treatment for before reading metacognitive strategies indicated that the MCAG treatment had a powerful effect on subjects’ level of awareness of before reading metacognitive strategies. ES values for MSI-D (ES = 0.13) and MSI-A (ES = 0.11) reflected the fact that during reading and after reading metacognitive

b

a

7.25 2.95 2.48 1.81 1.81 0.67 1.17 0.99 1.33 1.27 2.47

2.94 1.64 1.36 1.15 1.14 0.47 0.7 0.72 1.12 0.86 1.60

Control group 1 (N = 102) Mean SD

Research group

7.71 3.14 2.60 1.96 1.65 0.79 1.26 1.16 1.39 1.46 2.68

2.87 1.74 1.26 1.03 2.35 0.41 0.9 0.75 1.23 0.84 1.65

Content instruction group 2 (N = 85) Mean SD 9.14 4.54 2.66 1.94 1.28 0.87 1.65 1.33 1.74 1.21 3.60

3.16 1.81 1.38 1.18 1.12 0.34 0.83 0.74 1.26 0.83 1.91

Metacognitive treatment group 3 (N = 113) Mean SD 11.67 26.71 0.53 0.50 9.91 6.47 10.49 5.6 3.65 2.17 12.81

F2.297

Post hoc comparisons were performed using the Sidak t-test for multiple comparisons. Metacognitive Awareness Guidance

MSI-T MSI-B MSI-D MSI-A MSI-1 MSI-2 MSI-3 MSI-4 MSI-5 MSI-6 MSI-BCD

MSI parts

Table 3 Results of analysis of the MSI Questionnaire

0.0001 0.0001 0.59 0.607 0.0001 0.0018 0.0001 0.0041 0.0271 0.116 0.0001

p-value

3⬎1,2

3⬎1,2 3⬎1 3⬎1,2 3⬎1 3⬎1

3⬎1,2 3⬎1,2

Significant post hoc analysesa

0.62 0.92 0.13 0.11 0.48 0.49 0.62 0.46 0.34 –0.07 0.64

MCAGb group vs Control group

Effect size (ES)

646 E. Guterman / Learning and Instruction 13 (2003) 633–651

E. Guterman / Learning and Instruction 13 (2003) 633–651

647

strategies were not part of the MCAG treatment, and this is validated by the lack of effect found in the analysis. 5.4. The linkage between the three components of the assessment triangle— reading, metacognitive awareness and metacognitive strategies The analysis supports the main assumption of the study, that subjects who received metacognitive awareness guidance reveled a higher level of achievement on the three assessment tasks, and that increasing learners’ metacognitive awareness by integrating metacognitive learning principles into the reading assessment tasks also increases their chances of internalizing the guidance components and applying them in other learning activities. Engaging the learners in metacognitive awareness guidance affected their performance on assessment tasks—they attained a higher level of achievement on these tasks. Learners’ achievements on assessment tasks were significantly and positively correlated with their level of achievement on the MCAG. Learners who received metacognitive awareness guidance also demonstrated a significantly higher level of awareness of metacognitive reading strategies. It was assumed that subjects who performed better on the reading assessment tasks would also have higher scores on the MCAG questions. A Pearson test performed to determine whether there was any relationship between these variables revealed that subjects’ achievements on the three reading assessment tasks, on each separately and on all together, were significantly and positively related to subjects’ level of performance on the MCAG (Table 4). Table 4 Correlation matrix for achievement on reading assessment tasks and MCAG questionsa (N=113) ‘Paper’

‘Paper’ ‘End’

‘End’

MCAG-2

– ∗∗ 0.30 ∗∗ 0.28 ∗∗∗ 0.72 ∗∗∗ 0.63 NA

MCAG-3

NA

∗∗∗ 0.37 NA

MCAG-M

∗∗∗ 0.59

∗∗∗ 0.40

‘Dream’ All tasks MCAG-1

a

‘Dream’

All tasks

MCAG-1 MCAG-2 MCAG-3 MCAGM

– ∗∗ 0.28 ∗∗∗ 0.56 NA

– ∗∗∗ 0.57 NA NA ∗∗ 0.23 ∗∗∗ 0.32

∗ p⬍0.05, ∗∗ p⬍0.01, ∗∗∗ p⬍0.001.

– ∗∗∗ 0.60 ∗∗∗ 0.50 ∗∗∗ 0.43 ∗∗∗ 0.64

– ∗∗∗ 0.52 ∗∗∗ 0.53 ∗∗∗ 0.78

– ∗∗∗ 0.50 ∗∗∗ 0.88

– ∗∗∗ 0.73

–

648

E. Guterman / Learning and Instruction 13 (2003) 633–651

Since interrelationships were identified and validated between the MCAG and the tasks, it was assumed that the same interrelationship would be revealed between the MSI and the MCAG. Surprisingly, no significant correlation was identified between subjects’ level of achievement on the MCAG and their level of awareness of metacognitive reading strategies, as shown in Table 5. The results of this study showed a clear effect of written MCAG on performance on assessment tasks, and learners who engaged in metacognitive awareness guidance later demonstrated a significantly higher level of awareness of metacognitive reading strategies. However, students’ responses to the MCAG questions were found to be unsatisfactory, and no relation was found between the quality of their answers and their level of awareness of metacognitive reading strategies. If learners did not give satisfactory responses to the questions, what is it about the MCAG questions that can explain their effect? Clearly, it is not necessarily the quality of students’ responses on the MCAG that made the difference in their level of achievement on the reading tasks or on their level of awareness according to the MSI questionnaire. This question will be examined below.

6. Discussion This study was motivated by Vygotszky’s notion that we can extend and facilitate learners’ cognitive abilities and performance by providing a “more capable peer” (Vygotsky, 1978). This refers to a key concept within Vygotsky’s theory, the ‘zone Table 5 Correlation matrix for MCAG questions and MSIa MCAG-1 MCAG-2 MCAG-3 MCAG- MSIQ-T MSI-B M MCAG-1 MCAG-2

MSIQ-T MSI-B

– ∗∗∗ 0.52 ∗∗∗ 0.33 ∗∗∗ 0.78 0.03 0.02

∗∗∗ 0.50 ∗∗∗ 0.88 0.04 –0.00

∗∗∗ 0.73 0.07 0.05

0.04 0.02

MSI-D

0.03

0.01

0.05

0.04

MSI-A

0.01

0.12

0.06

0.08

MSI-BCD

0.00

0.01

–0.03

–0.00

MCAG-3 MCAG-M

a

MSI-D

MSI-A

MSIBCD

– –

∗ p⬍0.05, ∗∗ p⬍0.01, ∗∗∗ p⬍0.001.

– – ∗∗∗ 0.79 ∗∗∗ 0.72 ∗∗ 0.68 ∗∗∗ 0.89

– ∗∗ 0.27 ∗∗ 0.28 ∗∗∗ 0.78

– ∗∗∗ 0.34 ∗∗∗ 0.45

– ∗∗∗ 0.68

–

E. Guterman / Learning and Instruction 13 (2003) 633–651

649

of proximal development’ and views the written MCAG as a “psychological tool” which can empower and enrich students’ level of performance. The findings suggest that written MCAG does facilitate students’ performance on cognitive tasks. What seems to be significant is the role of the MCAG in creating engagement between the students and the reading process. From the principles and methods of implementation of the MCAG as described above, we can see that the students were guided to construct their own responses, use context information, focus on what they already know and understand, express opinions, anticipate, raise questions, suggest possible answers, use direct explicit self-talk, and reflect on what they know and how they can use it. We can therefore say that the MCAG created commitment, involvement, connection, obligation and responsibility. It weakened the detachment between the task and the pupil. They may not have known how to answer the questions, but it seems that their engagement with the task was the key to their higher level of achievement. The MCAG seems to act as a trigger to involvement and commitment to the process of reading and to function as a bridge between learning and cognitive development. Even when pupils do not exhibit a high level of activating prior knowledge, the process of activating, creating, and discovering prior knowledge seems to make the difference in their outcomes. The role of engagement in learning was described by one of the teachers interviewed: “Because these kinds of questions don’t have only one correct answer, all the pupils want to read their answers…. If I don’t let pupils present, they are hurt and offended. Because for some reason they are more attached to that kind of assignment” (Guterman, 2000 op. cit.; Guterman, 2002). Students’ engagement was enhanced through the MCAG which encouraged pupils to take responsibility for their performance by employing “active monitoring and consequent regulation and orchestration” of cognitive process to achieve cognitive goals (Flavell 1976, p. 282). This research points to student engagement as one of the intrinsic factors that effects their performance and level of achievement, and calls for using the MCAG to raise learners’ level of engagement in order to improve their performance and achievements. The utilization of MCAG in assessment still needs to be closely examined, elaborated and extensively negotiated among teachers, educators, assessment experts and researchers. Questions still remain: How and what to grade? Should there be a rating scale and, if so, how can rubrics be made sensitive enough to embrace the variety and diversity of pupils’ responses? Maybe the MCAG would be better assessed using new and different assessment methods such as interview, small group discussion, peer- or self-assessment. Or perhaps it should not be graded at all? This range of issues needs to be carefully and sensitively addressed. The ZPD is one of the more exciting concepts in cognitive development as it perceives the individual as an active agent in his or her own cognitive development. Individuals can achieve some of their goals by recruiting means already in their repertoire, and can be taught the skills needed to reach other goals. The field between the two represents the zone of proximal development. The findings of this study provide initial evidence to suggest that the MCAG utilized in this study engaged the

650

E. Guterman / Learning and Instruction 13 (2003) 633–651

students in actions within the ZPD and through this process enabled them to transcend their present level of performance and to approach their potential level. Incorporating MCAG into assessment tasks enabled students to improve their performance and to raise their level of awareness of metacognitive learning strategies. The MCAG affords learners the opportunity to engage in higher order operations: to test their knowledge, discover new links, anticipate, raise questions and suggest possible answers. As such, it functions within the ZPD, addresses the learners’ potential level of development and leads to internalization of the guidance. Thus, it not only facilitates their learning and improves outcomes on specific assessment tasks, but also increases the likelihood that they will apply higher order operations to other learning situations. It is possible that the MCAG acts as a process and that its effect is unidirectional. We can examine its effects and influence on students’ achievement and outcomes only when we consider the students’ level of metacognitive awareness at the outset of the research. We can hypothesize that in a second phase of the research (with the same subjects, but different reading assessment tasks), the subjects will demonstrate a higher level of metacognitive awareness, relative to the level found in this study. Finally, the combination of assessment and teaching applies to educators: it not only aims to extend and facilitate the development of students’ cognitive ability, but also provides a way to do so. It thus has the potential to improve learning, teaching, and assessment.

References Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading. In P. D. Pearson (Ed.), Handbook of reading research (pp. 255–292). New York: Longman. Anderson, R. C. (1983). Role of the reader’s schema during comprehension, learning, and memory. In R. C. Anderson, J. Osborn, & R. Tierney (Eds.), Learning to read in American schools (pp. 85–121). Hillsdale, NJ: Erlbaum. Anderson, R. C., Reynolds, R. E., Shallert, D. L., & Goetz, E. T. (1977). Frameworks for comprehending discourse. American Educational Research Journal, 14, 367–382. Anderson, T. H., & Armbruster, B. B. (1980). Studying. Reading Research Quarterly, 14, 605–623. Anderson, T. H. (1980). Study strategies and adjunct aids. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 483–503). Hillsdale, NJ: Erlbaum. Baumann, J. F., & Schmitt, M. C. (1987). The what, why, how, and when of comprehension instruction. The Reading Teacher, 39, 640–646. Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Erlbaum. Bransford, J. D., & Johnson, M. K. (1972). Contextual prerequisites for understanding: Some investigation of comprehension and recall. Journal of Verbal Learning and Verbal Behavior, 11, 717–726. Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.). (1999). How people learn: Brain, mind, experiences, and school. Washington, DC: National Research Council—National Academy Press. Bransford, J. D., Stein, B. S., Arbitman-Smith, R., & Vye, N. J. (1985). Three approaches to improving thinking and learning skills. In J. Segal, S. Chipman, & R. Glaser (Eds.), Thinking and learning skills: Relating instruction to basic research (vol. 1) (pp. 93–135). Hillsdale, NJ: Erlbaum. Bransford, J. D., Stein, B. S., Shelton, T. S., & Owings, R. A. (1980). Cognition and adaptation: The importance of learning to learn. In J. Harvey (Ed.), Cognition, social behavior and the environment. Hillsdale, NJ: Erlbaum. Braun, C., Rennie, B. J., & Labercane, G. D. (1986). A conference approach to the development of

E. Guterman / Learning and Instruction 13 (2003) 633–651

651

metacognitive strategies. In Solving problems is literacy: Learners, teachers, and researchers (pp. 204-209). The 35th Yearbook of the National Reading Conference. Brown, A. L., & French, L. A. (1979). The zone of potential development: Implications for intelligence testing in the year 2000. Intelligence, 3, 255–273. Brown, A. L., & Ferrara, R. A. (1985). Diagnosing zones of proximal development. In J. V. Wertsch (Ed.), Culture, communication, and cognition: Vygotskian perspectives (pp. 273–305). Cambridge: Cambridge University Press. Brown, A. L., Campione, J. C., & Barclay, C. R. (1979). Training self-checking routines for estimating test readiness: Generalization from list learning to prose recall. Child Development, 50, 501–512. Brown, A. L., Campione, J. C., Reeves, R. C., & Ferrara, R. A. (1990). Interactive learning, individual understanding and formal schooling. In L. Tolchinsky (Ed.), Culture, schooling, and psychological development (pp. 129–161). Norwood, NJ: Ablex. Brown, A. L., Smiley, S. S., & Lawton, S. C. (1978). The effects of experience on the selection of suitable retrieval cues for studying texts. Child Development, 49, 829–835. Campione, I. C., Brown, A. L., Reeve, R. A., Ferrara, R. A., & Palincsar, A. S. (1991). Interactive learning and individual understanding: The case of reading and mathematics. In L. Tolchinsky-Landsmann (Ed.), Culture, schooling and psychological development (pp. 136–170). Norwood, NJ: Ablex. Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence (pp. 281–299). Hillsdale, NJ: Erlbaum. Guterman, E. (2002). Toward dynamic assessment of reading: Applying metacognitive guidance to reading assessment tasks. Journal of Research in Reading, 25(3), 304–319. Inbar, E. (1995). The assessment task collection of reading and writing: Principles, aims, and teacher instruction. The Israel Ministry of Education (in Hebrew). Lewy, A. (1997). Alternative assessment, theory and practice: A collection of cases and reflective comments. Tel-Aviv, Israel: Mofet Institute Press (Hebrew with English abstract). Lonberger, R. (1988). Effects of training in a self generated learning strategy on the prose processing abilities of 4th and 6th graders. Paper presented at the annual meeting of the Eastern Education Association, Savannah, GA. Moll, C. L. (1990). Vygotsky and education. New York: Cambridge University Press. Neisser, U. (1976). Cognition and reality: Principles and implications of cognitive psychology. San Francisco, CA: Freeman. Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension- fostering and comprehension-monitoring activities. Cognition and Instruction, 1, 117–175. Pellegrino, J.W., Chudowsky, N., & Glaser, R. (Eds.). (2002). Knowing what students know: The science and design of educational assessment. Washington, DC: National Research Council—National Academy Press. Riscko, V. J., & Feldman, N. (1986). Teaching young remedial readers to generate questions as they read. Reading Psychology, 23, 54–64. Rumelhart, D. E. (1980). Schemata: The building blocks of cognition. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 33–58). Hillsdale, NJ: Erlbaum. Schmitt, M. C. (1988). The effects of an elaborated directed activity on the meta-comprehension skills of third graders. In Dialogues in literacy research (pp. 167-181). The 37th Yearbook of the National Reading Conference. Schmitt, M. C. (1990). A questionnaire to measure children’s awareness of strategic reading processes. The Reading Teacher, 43(3), 454–461. Spiro, R. J., & Tierre, W. C. (1980). Individual differences in schema utilization during discourse processing. Journal of Educational Psychology, 72, 204–208. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Cambridge, MA: Harvard University Press. Wertsch, J. V. (1985). Vygotsky and the social formation of mind. London: Harvard University Press. Wilson, P. T., & Anderson, R. C. (1986). What they don’t know will hurt them: The role of prior knowledge in comprehension. In J. Orasanu (Ed.), Reading comprehension: From research to practice (pp. 31–49). Hillsdale, NJ: Erlbaum.