- Email: [email protected]
Comprehension monitoring skills of adults with mental retardation
Research in Developmental Disabilities, Vol. 14, pp. 409-421, Printed in the USA. All rights reserved.
1993
0891-4222/93 $6.00 + .OO Copyright Q 1993 Pergamon Press Ltd.
Comprehension Monitoring Skills of Adults With Mental Retardation Martin Fujiki and Bonnie Brinton Brigham Young University
The ability of 30 adults with mild and moderate levels of mental retardation to monitor their comprehension while performing a direction-following rask was examined. Subjects were “employed” to aid the investigator in compiling objects for gifr bags. The task consisted of 60 directions (e.g., “Give me a blue pencil”), of which 6 contained trouble sources (ambiguous directions, unintelligible words, and compliance problems). Each subject’s response to the directions involving trouble sources was scored to determine if the subject (a) demonstrated ef/ective comprehension monitoring (as b&cared by an immediate awareness of the problem and effective attempt to rectify the problem), (b) requested clarification after attempting to comply with the direction and being unable to do so, (c) demonstrated ineffective comprehension monitoring (as indicated by an awareness of the problem bur an ineflective means of dealing with it), or (d) showed no awareness of the trouble source. The abiliv of subjects to monitor comprehension varied with type of trouble source. Ambiguous trouble sources were the most dificult for the subjects to detect, and compliance-problem trouble sources were the most frequently ide@ied.
Successful language comprehension involves the processing of incoming information at two levels (Flavell, 1981). On one level, the listener’s task is to understand the message that a speaker is attempting to communicate. At
This research was in part supported by Grant no. H133C90121 from the National Institute on Disability and Rehabilitation Research, Department of Education, to the University of Kansas Parsons Research Center. The authors wish to acknowledge CLASS LTD and the Parsons State Hospital and Training Center for assistance in subject identification and selection. They also wish to acknowledge the Undergraduate Research Trainee program at Brigham Young University and Ms. Valyne Watson for her help in subject selection and data coding in the social validation aspect of the study. Requests for reprints should be sent to Martin Fujiki, Speech Language Pathology, Brigham Young University, 130 TLRB, Provo, UT 834602. 409
410
M. Fujiki and B. Brinton
the same time, on another level, the listener must monitor whether incoming messages are understood and detect points at which any comprehension problems occur. This second level of processing is important because interactive exchanges are replete with potential trouble sources that may result in communication breakdown. Environmental distraction, lack of shared background information, and ambiguous reference are only three examples of problems that might hinder the comprehension of a linguistic message. Once a listener detects a breakdown in comprehension, various strategies may be employed to repair the misunderstanding. The ability to detect and respond to breakdowns in one’s own understanding of language has been referred to as comprehension monitoring (Markman, 1981). Although comprehension and comprehension monitoring are closely related, it is important to note that they are not identical behaviors (Dollaghan, 1987). As Dollaghan observed, a listener is first and foremost attempting to comprehend the message that the speaker has produced. In contrast, comprehension monitoring refers to the listener’s awareness of whether or not the message has been understood. As such, comprehension monitoring is a metacognitive process that begins to emerge early in life and continues developing into adolescence. Normally developing children as young as 2 and 3 years of age have some ability to monitor comprehension (Revelle, Welhnan, & Karabenick, 1985). By age 6 or 7 years, children are able to consistently monitor their understanding of language in many contexts (Bonitatibus, 1988). However, development continues after this time (e.g., Flavell, Speer, Green, & August, 1981; Markman, 1977), with some highly complex tasks like recognizing contradictory elements in spoken text not mastered until as late as age 12 (e.g., Markman, 1979). In considering the development of comprehension monitoring, there are numerous message and listener-related variables that may influence an individual’s performance (see Dollaghan, 1987, for review). Thus, general age-based estimates of mastery must be considered with caution. The capability to monitor one’s own understanding may be particularly critical for persons with impaired language skills. These individuals not only have a higher frequency of comprehension problems to resolve when compared to nonimpaired peers, but may also have difficulty with the task of comprehension monitoring itself (see Dollaghan & Kaston, 1986; Skarakis-Doyle, MacLellan, & Mullin, 1990; Skarakis-Doyle & Mullin, 1990 for discussion). In this context, persons with mental retardation may be particularly susceptible to difficulties with comprehension monitoring. Even relatively high-functioning individuals with mental retardation may exhibit a variety of comprehension problems in comparison to their nonmentally retarded peers (Chapman & Nation, 1981). Further, for some individuals, these problems may extend beyond what might be expected given performance in nonverbal problem solving domains (Abbeduto, Furman, &
Comprehension Monitoring
411
Davies, 1989). In conjunction with these findings, it must also be recognized that persons with mental retardation demonstrate limited cognitive resources that affect a wide range of behaviors (Brooks, Sperber, & McCauley, 1984). Taken together, these factors would suggest that some individuals with mental retardation would experience a high rate of comprehension breakdown and would be less likely to detect and resolve these breakdowns when they occur. In one of the few studies examining the comprehension-monitoring skills of persons with mental retardation, Ezell and Goldstein (1991) employed a screening procedure to determine if children with mild to moderate mental retardation monitored their comprehension (as indicated by the failure to request clarification when presented with inadequate instructions). The 23 children screened requested clarification an average of 6 times each out of a possible 20 trials. As the authors noted, this finding suggested that the children had difficulty detecting comprehension problems and/or requesting clarification of inadequate messages. Following this screening procedure, Ezell and Goldstein (1991) used an observational learning paradigm to facilitate comprehension-monitoring skills in five of the children. These subjects, who ranged in age from 7:lO years to 10:8 years, generally demonstrated good success on this training program. Results such as those presented by Ezell and Goldstein suggest that further research investigating comprehension monitoring in individuals with mental retardation is warranted. For children and adults in many academic and work settings, detecting and requesting clarification of messages that are not understood could be critical. Following verbal instructions, responding to questions, and conveying information from one person to another all involve language comprehension. The failure to consistently identify and resolve comprehension problems in any of these situations may seriously jeopardize employment. For example, consider the importance of recognizing and responding to misunderstood directions in a work setting such as a fast food restaurant. If an individual is assembling a pizza and receives a direction with an unfamiliar referent (such as “anchovies,“) that individual must request clarification of the troublesome element (e.g. “I don’t understand,” “What’s that?’ or “What’s an anchovy?“). Failure to detect the gap in comprehension, or failure to act on the gap if it is detected, is likely to have unfortunate consequences. Treatment programs to facilitate comprehension monitoring of verbal directions could be helpful for individuals with mental retardation to prepare them for independent lifestyles. However, before such programs can be designed and implemented, it is necessary to investigate how individuals with mental retardation behave on tasks that highlight comprehensionmonitoring skills. This investigation addressed this need by examining the
412
M. Fujiki and B. Brinton
ability of adults with mild to moderate levels of mental retardation to demonstrate comprehension-monitoring skills in a structured work task. Subjects were presented with trouble sources interspersed throughout a series of verbal directions from an employer. This particular setting was selected because of its importance for persons with mild and moderate levels of mental retardation who are working in community settings. Because it is necessary not only to detect but also to resolve these comprehension problems (e.g., by requesting clarification), the specific strategies used to signal comprehension problems were of particular interest. METHOD Subjects
Thirty subjects (13 men and 17 women) participated in the study. Nineteen subjects were sampled from community placements, and 11 were sampled from an institutional placement to provide a wide sampling of the population with mental retardation. Subjects were sampled from a larger group of persons with mental retardation to meet the following criteria: 1. Full scale Wechsler Adult Intelligence Scale-Revised (WAIS-R) (Wechsler, 1981) IQ within the mild to upper moderate range of mental retardation, The resulting group produced a mean WAIS-R full scale IQ of 61.43 (SD = 7.40). 2. Chronological age between 20 and 42 years. The resulting group produced a mean age of 28.9 years (SD = 6.27). 3. Auditory and visual status within normal limits as demonstrated by passing standard vision and hearing screening examinations. Normal visual status was defined as current binocular vision of 20/40 or better (with corrective lenses if needed). Normal auditory status was defined as current pure tone threshold of 35 dB or lower through the speech frequencies in the better ear. 4. Ambulatory, as demonstrated by the ability to walk independently. 5. Adequate verbal language skills to participate in conversation. All subjects produced a mean length of utterance, in morphemes, of over 4.0. Procedures
Each subject was assessed individually by the same examiner. Community-based subjects were seen at their supported work settings. Institutionalized subjects were seen in a research facility on the campus of the institution. All of these subjects were familiar with the research facility and had participated in previous studies. To motivate subjects to perform, the following procedure was used: Each subject initially met the investiga-
Comprehension Monitoring
413
tor in a situation structured as a job interview. During this 30-min interview, the investigator explained the task to be performed and told each subject that if they agreed to perform the required work they would be paid $3.30/h for their time (minimum wage at the time of the study). Although it was stressed that participation was voluntary, all subjects agreed to participate. Following the interview session, the trial phase of the study was initiated. Four containers were placed on a table in front of the subject. Each container was filled with colored paper clips, combs, rubber bands, or pencils. Each subject was asked to help the investigator fill gift bags with various combinations of these items. Subjects were initially presented with the following directions: I need to fill some gift bags. Each bag will have something different in it. I’d like you to help me find the things that go in each bag. I’ll tell you what I need and you give it to me, ok? Be sure to let me know if you don’t understand something.
Following these instructions, subjects performed four trial items, designed to insure that they were familiar with each of the objects and colors to be used in the task (e.g., Give me a red pencil, give me a green rubber band, etc.). If the subject performed all of the trial items appropriately, the experimental task was initiated. If any of the trial items was missed, further trial items were administered. It was necessary to administer additional trial items to only one of the 30 subjects who participated in the study. Another indication of the ability of the subjects to perform the direction-following task was the fact that 97% of the control directions (those that did not contain trouble sources) were completed appropriately on the first presentation. Control directions that were performed inappropriately were readministered a second time. The experimental task consisted of a set of 60 directions, 6 of which involved trouble sources (2 of each of 3 types of trouble sources), to represent 10% of the total number of directions. Directions containing trouble sources were randomly distributed. The three types of trouble sources used are described as follows: 1. Unintelligible (nonsense) word. The investigator substituted an unintelligible word in place of a key word (e.g., “Give me the blue /kliv/” and “Give me the green /gLIb/“). 2. Ambiguous referent. The investigator asked the subject for an item without providing enough detail to indicate the specific object desired (e.g., “Give me a rubber band” when there were several different colors of rubber bands available). 3. Compliance problem. The investigator requested that the subject act on an object that was not present (e.g., “Give me a red toothbrush” when no toothbrushes were present).
414
M. Fujiki and B. Brinton
The task took approximately 20 min to complete. The entire set of trouble source directions, examples of the control directions, and a more detailed explanation of administration procedures are available from the first author. Data Analysis All experimental procedures were videotape recorded. Each subject’s response to the directions involving trouble sources was coded into one of the four categories described here. Nonverbal behaviors, in addition to verbal behaviors, were used as indications of monitoring (Flavell et al., 1981; Skarakis-Doyle et al., 1990). The categories used reflect the view that comprehension monitoring must involve both evaluation and regulation (Ackerman, 1986; Baker, 1985). From this perspective, successful comprehension monitoring must involve both the detection of the comprehension problem (evaluation), and the attempt to resolve the problem through appropriate action (regulation). Specific categories of behavior were as follows: Effective comprehension monitoring 1. The subject immediately requested clarification of the direction, either verbally (e.g., “Huh?’ or “Could you repeat that?‘) or nonverbally (e.g., subject made a distinct gesture to the investigator as if to say, “What?‘). 2. The subject briefly (2 s or less) hesitated or glanced at the materials and then requested clarification. The subject did not visually scan (as indicated by looking from side to side) or physically search for the requested object. 3. The subject requested clarification in the process of reaching for an item (e.g., The subject reached for an item and at the same time said, “This?“). In these cases, the subject immediately reached for a single item without physically searching or shuffling through items. Request for clarification following attempt to comply. These responses also represented appropriate strategies for dealing with the comprehension problem but did not represent the same level of efficiency as the strategies discussed previously. “Effective monitoring” represented an immediate detection and subsequent action to resolve the trouble source. Responses categorized as “request for clarification following attempt to comply” represented detection of the trouble source only after the subject attempted to follow the direction and could not do so. (e.g., Revelle et al., 1985). 1. The subject physically touched or shuffled through the materials (longer than 2 s), then requested clarification. Because the 2-s delay was somewhat arbitrary, this criterion was always paired with other physically
Comprehension Monitoring
415
observable behaviors (e.g., physically searching through the materials) in categorizing the subject’s performance. 2. The subject visually scanned (by looking from side to side) the materials (longer than 2 s) and then requested clarification. 3. The subject reached for an item, stopped, and then requested clarification. Ineffective monitoring 1. The subject produced an immediate nonverbal reaction to the direction (e.g., suddenly tensing the body) but did not request clarification. This category also included cases in which a subject laughed upon hearing the direction. In all cases, the nonverbal response was followed by the subject’s handing the investigator an item, apparently guessing as to what was being requested. 2. The subject searched through the objects physically or scanned the objects visually for longer than 2 s, suggesting that there was some awareness that the direction could not be followed. However, rather than requesting clarification, the subject handed the investigator an item, again apparently guessing as to what was being requested. 3. The subject requested clarification of the wrong item in the sentence. For example, when asked for a blue /kliv/, a subject responded, “What color?’ No response to the trouble source 1. The subject handed the investigator an item without hesitation. The subject showed no response to the trouble source. Reliability Coding reliability was established using the following procedures. Following extensive training with the classification system, two investigators independently coded 20% of the data base using videotaped recordings of the data collection. The six subjects used to establish coding reliability were randomly selected from the sample. The percentage of agreement between investigators was calculated using the following formula: A/N x 100, where A was the total number of coding agreements, and N was the total number of observations to be coded. This procedure produced a percent agreement of 97%. Social Valia!ation In order to provide a measure of social validation, the same experimental procedure was also administered to a group of 23 adults. These individ-
416
A4. Fujiki and B. Brinton
uals were college students enrolled in a course in child language acquisition. Subjects participated on a volunteer basis and ranged in age from 18 to 34 years. These subjects appropriately monitored comprehension on 98% of the unintelligible items, 100% of the compliance problem items, and 85% of the ambiguous items. These results suggested that adults could be expected to identify the vast majority of trouble sources. However, it was of note that ambiguous trouble sources were the least powerful in eliciting a response. On several occasions (7 of 46), subjects responded to an ambiguous direction by simply handing the examiner an object. When debriefed and asked why they responded in this manner, these individuals noted that they thought that because no color was specified (e.g., “Give me a pencil”), the matter of color was their choice. RESULTS Statistical evaluation of the performance of the institutional and community subjects using unpaired t-tests on individual categories of performance indicated that group differences were not significant. Thus, the data from the groups were combined for further analyses. Table 1 presents the proportion of responses falling into each of the four basic response categories of (a) effective monitoring, (b) request for clarification following attempt to comply, (c) ineffective monitoring, and (d) no response to trouble source. For all three trouble sources, effective monitoring was the most frequently observed response. However, the proficiency with which subjects monitored trouble sources varied according to type of trouble source. As Table 1 illustrates, subjects identified and requested clarification of complianceproblem trouble sources at the highest rate of success, followed by unintelligible and ambiguous trouble sources.
TABLE 1 Response Patterns Used by Adults With Retardation
to Three Types of Trouble Sources Trouble Source
Effective monitoring Request for clarification following attempt to comply Ineffective monitoring No resuonse
Unintelligible
Ambiguous
Compliance
47% (28)
37% (22)
58% (35)
25% (15) 10% (6) 18% (11)
30% (18) 8% (5) 25% (15)
25% (15) 15% (9) 2% (1)
Note. Numbers in parentheses represent raw frequency totals.
417
Comprehension Monitoring
As a group, subjects identified compliance-problem trouble sources most readily. This observation was supported by analysis of the appropriate performance of the subjects. The two categories that represented appropriate performance were combined (“effective monitoring” and “request for clarification following attempt to comply”), and a one-way ANOVA with repeated measures was employed to determine if subject performance differed between trouble sources. Differences in performance between types of trouble sources were evident [F(2, 87) = 6.47, p < .Ol]. Subsequent post hoc analysis using the Newman-Keuls’ Test indicated that the ambiguous trouble sources were significantly more difficult than the compliance trouble sources (p c .Ol). Several different strategies of performance were included in the categories of “request for clarification following attempt to comply” and “ineffective monitoring.” The specific strategies grouped in each of these categories are presented in Tables 2 and 3, respectively. Table 2 presents specific strategies categorized as “request for clarification following attempt to comply.” These responses were considered as appropriate, but somewhat less efficient than responses categorized as “effective monitoring.” Subjects most frequently attempted to physically or visually search for the item, and then requested clarification of the direction. However, in a few cases, subjects reached for an item, stopped, and then requested clarification. It should be considered that although the observed searching behaviors were interpreted as attempts to comply with the direction, they may have been stalls for more processing time to deal with the problematic direction. Table 3 details responses categorized as “ineffective monitoring.” On 8-15% of the trials for the three types of trouble sources the subjects gave some nonverbal indication that they realized there was a problem, but then failed to request clarification. On most of these trials, the subTABLE 2 Strategies Classified as Request for Clarification
Following an Attempt to Comply
Unintelligible Overall percentage and frequency of occurrence Strategies Verbal clarification request following physical search or visual scan Verbal clarikation request following reach for item
Ambiguous
Compliance
25% (15)
30% (18)
25% (15)
22% (13)
27% (16)
25% (15)
3% (2)
3% (2)
0%
(0)
Note. Percentages represent proportion of all responses to each trouble source type. Numbers in parentheses represent raw frequency totals.
418
A4. Fujiki and B. Brinton
TABLE 3 to Represent “Ineffective Monitoring”
Strategies Considered
Overall percentageand frequency of occurfence Strategies Supplying item following search Nonverbal indication of awareness (no request for clarification) Requesting clarificationof wrong item
Unintelligible
Ambiguous
Compliance
10%(6)
8% (5)
15%(9)
5% (3)
0% (0)
8% (5)
5% (3) 0% (0)
1% (4) 2%a(1)
1% (4) 0% (0)
Note. Numbers in parenthesesrepresentraw frequency totals.
aPercentagesdo not sum to 8 because numbersare roundedto nearest whole.
jects eventually attempted to comply with the direction by handing the investigator an item. For example, one subject suddenly tensed upon hearing the direction but did not request clarification. Instead, she handed the investigator an item, apparently guessing as to what was wanted. Another subject consistently looked at the materials, looked up at the investigator, and then looked at the materials again without requesting clarification.
DISCUSSION
Subjects with mental retardation requested repair of the majority of trouble sources, either immediately or after an attempt to comply (74% overall). However, for over one-fourth of the inadequate directions, subjects either did not detect the trouble sources or did not provide feedback regarding the presence of the trouble sources to the investigator. In these cases, subjects appeared to guess at the requested item, either immediately (within 2 s) or following a scan or search for the item. In one case, a subject requested repair of an element that was intelligible while making no comment on the troublesome element (e.g. “What color?’ in response to “Give me a blue [kliv]“); however, most of the time that subjects did not respond to trouble sources, they attempted to follow the directions despite the trouble sources. It was clear that comprehension monitoring was influenced by the types of trouble sources embedded in the verbal directions. Subjects requested clarification of unintelligible directions and compliance-problem directions more often than they requested clarification of ambiguous directions. This result was not surprising considering the performance of the subjects on the social validation measure used and the performance of children on similar tasks as described in the literature (e.g., Revelle et al., 1985).
Comprehension
Monitoring
419
There are at least two possibilities to explain the particular difficulty some subjects experienced with ambiguous trouble sources. First, it is possible that for some individuals, the context of the task did not provide strong enough cues to demonstrate that both object and color were important in item selection. Subjects might then proceed as if the color of the object were optional. This was evidently the case for a few of the subjects without disabilities. However, it is also possible, in light of the subtlety of the trouble sources, that some subjects may not have detected the ambiguity. Regardless of the factors that influenced the subjects’ performance, the comprehension monitoring demonstrated in this task as a whole was less accurate than might be hoped considering the importance of the behaviors assessed to an individual’s functioning within a community. Most, if not all, supported and independent employment settings in which individuals with mental retardation work demand that employees follow verbal directions as accurately as possible. It is important to request clarification of unclear directions before completing the commands. Guessing at unclear elements will usually result in an error that detracts from the employee’s productivity. Although it can be argued that the subjects in this study detected more trouble sources than they signaled verbally, in a practical sense th,s lnade little difference in how efficiently they would function in a work setting. The findings of this study suggest several clinical implications. First, requesting repair of genuine misunderstandings in work tasks is an important skill that may warrant attention in intervention. It is likely that some individuals with mental retardation may have difficulty detecting and resolving comprehension problems beyond that which might be expected considering their general levels of functioning. Treatment programs designed to prepare and support individuals with mental retardation in securing and maintaining employment in community settings might include strategies for handling miscommunications. Requesting clarification of trouble sources might be facilitated in the context of improving work productivity and accuracy. Specifically, clinical treatment programs could evaluate and subsequently treat (if indicated) recognition of instances where comprehension of important information is jeopardized. With regard to recognizing ambiguity in verbal instructions, detection of subtle trouble sources could be stressed by comparing “What do I know about this task?’ with “What do I need to know about this task?‘. Strategies for requesting clarification in efficient, socially acceptable ways could be suggested. Although immediate recognition and request for repair constitutes the most efficient manner of resolving miscommunications, requesting repair after a failed attempt to complete a task could also be a viable method. Finally, acknowledging and acting upon repairs elicited could constitute the final step in treatment.
420
A4. Fujiki and B. Brinton
Repairing trouble sources in interaction is an interdependent behavior negotiated between conversational partners. Even though the behavior itself is interdependent, the ability to initiate a repair sequence is a skill that facilitates social independence. Knowing how and when to request help can widen the social sphere in which one can function. In conclusion, it should be noted that despite the ecological validity of the task, it is not clear whether the performance observed in these subjects was representative of behavior in employment settings. Further research in actual employment settings is needed to clarify this issue. In addition, it is not clear whether subjects failed to request clarification because of social reasons (e.g., fear of questioning the investigator) or whether they actually were unaware of the trouble source. Both of these issues should be explored in further investigations. REFERENCES Abbeduto, L., Furman, L., & Davies, B. (1989). The relationship between receptive language and mental age in persons with mental retardation. American Journal on Mental Retardation, 93, 535-543.
Ackerman, B. P (1986). Children’s sensitivity to comprehension failure in interpreting a nonliteral use of an utterance. Child Development, 57,485-497. Baker, L. (1985). How do we know when we don’t understand? Standards for evaluating text comprehension. In D. Forrest-Pressley, G. E. MacKinnon, & G. T. Wailer (Eds.), Metacognition, cognition and human performance (Vol. I. pp. 155-205). Orlando, FL: Academic Press. Bonitatibus, G. (1988). Comprehension monitoring and the apprehension of literal meaning. Child Development, 59,60-70.
Brooks, I? H., Sperber, R., &t McCauley, C. (1984). Learning and cognition in the mentally retarded. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Chapman, D. L., & Nation, J. E. (1981). Patterns of language performance in educable mentally retarded children. Journal of Communication Disorders, 14.245-254. Dollaghan, C. (1987). Comprehension monitoring in normal and language-impaired children. Topics in Language Disorders, 7,45-60.
Dollaghan, C., & Kaston, N. (1986). A comprehension monitoring program for languageimpaired children. Journal of Speech and Hearing Disorders, 51,264-271. Ezell, H. K., & Goldstein, H. (1991). Observational learning of comprehension monitoring skills in children who exhibit mental retardation. Journal of Speech and Hearing Research, 34. 141-154.
Flavell, J. (1981). Cognitive monitoring. In W. P. Dickson (Ed.), Children’s oral communication skills (pp. 35-60). New York Academic Press. Flavell, J., Speer, J., Green, F., & August, D. (1981). The development of comprehension monitoring and knowledge about communication. Monographs of the Society for Research in Child Development, No. 192.
Markman, E. (1977). Realizing that you don’t understand: A preliminary investigation. Child Development, 48,986-992.
Markman, E. (1979). Realizing that you don’t understand: Elementary school children’s awareness of inconsistencies. Child Development, 50.643-655. Markman, E. (1981). Comprehension monitoring. In W. P. Dickson (Ed.), Children’s oral cornmunication skills (pp. 61-84). New York: Academic Press.
Comprehension Monitoring
421
Revelle, G. L., Welhnan, H. M., & Karabenick, J. D. (1985). Comprehension monitoring in preschool children. Child Developmen?, 56,654-663. Skarakis-Doyle, E., MacLellan, N., & Mullin, K. (1990). Nonverbal indicants of comprehension monitoring in language-disordered children. Journal of Speech and Hearing Disorders, 55, 461-467.
Skarakis-Doyle, E., & Mullin, K. (1990). Comprehension monitoring in language-disordered children: A preliminary investigation of cognitive and linguistic factors. Journal of Speech and Hearing Disorders, 55,700-705.
Wechsler, D. (1981). Wechsler Adult Intelligence Scale-Revised. Corporation.
Cleveland: Psychological
1993
0891-4222/93 $6.00 + .OO Copyright Q 1993 Pergamon Press Ltd.
Comprehension Monitoring Skills of Adults With Mental Retardation Martin Fujiki and Bonnie Brinton Brigham Young University
The ability of 30 adults with mild and moderate levels of mental retardation to monitor their comprehension while performing a direction-following rask was examined. Subjects were “employed” to aid the investigator in compiling objects for gifr bags. The task consisted of 60 directions (e.g., “Give me a blue pencil”), of which 6 contained trouble sources (ambiguous directions, unintelligible words, and compliance problems). Each subject’s response to the directions involving trouble sources was scored to determine if the subject (a) demonstrated ef/ective comprehension monitoring (as b&cared by an immediate awareness of the problem and effective attempt to rectify the problem), (b) requested clarification after attempting to comply with the direction and being unable to do so, (c) demonstrated ineffective comprehension monitoring (as indicated by an awareness of the problem bur an ineflective means of dealing with it), or (d) showed no awareness of the trouble source. The abiliv of subjects to monitor comprehension varied with type of trouble source. Ambiguous trouble sources were the most dificult for the subjects to detect, and compliance-problem trouble sources were the most frequently ide@ied.
Successful language comprehension involves the processing of incoming information at two levels (Flavell, 1981). On one level, the listener’s task is to understand the message that a speaker is attempting to communicate. At
This research was in part supported by Grant no. H133C90121 from the National Institute on Disability and Rehabilitation Research, Department of Education, to the University of Kansas Parsons Research Center. The authors wish to acknowledge CLASS LTD and the Parsons State Hospital and Training Center for assistance in subject identification and selection. They also wish to acknowledge the Undergraduate Research Trainee program at Brigham Young University and Ms. Valyne Watson for her help in subject selection and data coding in the social validation aspect of the study. Requests for reprints should be sent to Martin Fujiki, Speech Language Pathology, Brigham Young University, 130 TLRB, Provo, UT 834602. 409
410
M. Fujiki and B. Brinton
the same time, on another level, the listener must monitor whether incoming messages are understood and detect points at which any comprehension problems occur. This second level of processing is important because interactive exchanges are replete with potential trouble sources that may result in communication breakdown. Environmental distraction, lack of shared background information, and ambiguous reference are only three examples of problems that might hinder the comprehension of a linguistic message. Once a listener detects a breakdown in comprehension, various strategies may be employed to repair the misunderstanding. The ability to detect and respond to breakdowns in one’s own understanding of language has been referred to as comprehension monitoring (Markman, 1981). Although comprehension and comprehension monitoring are closely related, it is important to note that they are not identical behaviors (Dollaghan, 1987). As Dollaghan observed, a listener is first and foremost attempting to comprehend the message that the speaker has produced. In contrast, comprehension monitoring refers to the listener’s awareness of whether or not the message has been understood. As such, comprehension monitoring is a metacognitive process that begins to emerge early in life and continues developing into adolescence. Normally developing children as young as 2 and 3 years of age have some ability to monitor comprehension (Revelle, Welhnan, & Karabenick, 1985). By age 6 or 7 years, children are able to consistently monitor their understanding of language in many contexts (Bonitatibus, 1988). However, development continues after this time (e.g., Flavell, Speer, Green, & August, 1981; Markman, 1977), with some highly complex tasks like recognizing contradictory elements in spoken text not mastered until as late as age 12 (e.g., Markman, 1979). In considering the development of comprehension monitoring, there are numerous message and listener-related variables that may influence an individual’s performance (see Dollaghan, 1987, for review). Thus, general age-based estimates of mastery must be considered with caution. The capability to monitor one’s own understanding may be particularly critical for persons with impaired language skills. These individuals not only have a higher frequency of comprehension problems to resolve when compared to nonimpaired peers, but may also have difficulty with the task of comprehension monitoring itself (see Dollaghan & Kaston, 1986; Skarakis-Doyle, MacLellan, & Mullin, 1990; Skarakis-Doyle & Mullin, 1990 for discussion). In this context, persons with mental retardation may be particularly susceptible to difficulties with comprehension monitoring. Even relatively high-functioning individuals with mental retardation may exhibit a variety of comprehension problems in comparison to their nonmentally retarded peers (Chapman & Nation, 1981). Further, for some individuals, these problems may extend beyond what might be expected given performance in nonverbal problem solving domains (Abbeduto, Furman, &
Comprehension Monitoring
411
Davies, 1989). In conjunction with these findings, it must also be recognized that persons with mental retardation demonstrate limited cognitive resources that affect a wide range of behaviors (Brooks, Sperber, & McCauley, 1984). Taken together, these factors would suggest that some individuals with mental retardation would experience a high rate of comprehension breakdown and would be less likely to detect and resolve these breakdowns when they occur. In one of the few studies examining the comprehension-monitoring skills of persons with mental retardation, Ezell and Goldstein (1991) employed a screening procedure to determine if children with mild to moderate mental retardation monitored their comprehension (as indicated by the failure to request clarification when presented with inadequate instructions). The 23 children screened requested clarification an average of 6 times each out of a possible 20 trials. As the authors noted, this finding suggested that the children had difficulty detecting comprehension problems and/or requesting clarification of inadequate messages. Following this screening procedure, Ezell and Goldstein (1991) used an observational learning paradigm to facilitate comprehension-monitoring skills in five of the children. These subjects, who ranged in age from 7:lO years to 10:8 years, generally demonstrated good success on this training program. Results such as those presented by Ezell and Goldstein suggest that further research investigating comprehension monitoring in individuals with mental retardation is warranted. For children and adults in many academic and work settings, detecting and requesting clarification of messages that are not understood could be critical. Following verbal instructions, responding to questions, and conveying information from one person to another all involve language comprehension. The failure to consistently identify and resolve comprehension problems in any of these situations may seriously jeopardize employment. For example, consider the importance of recognizing and responding to misunderstood directions in a work setting such as a fast food restaurant. If an individual is assembling a pizza and receives a direction with an unfamiliar referent (such as “anchovies,“) that individual must request clarification of the troublesome element (e.g. “I don’t understand,” “What’s that?’ or “What’s an anchovy?“). Failure to detect the gap in comprehension, or failure to act on the gap if it is detected, is likely to have unfortunate consequences. Treatment programs to facilitate comprehension monitoring of verbal directions could be helpful for individuals with mental retardation to prepare them for independent lifestyles. However, before such programs can be designed and implemented, it is necessary to investigate how individuals with mental retardation behave on tasks that highlight comprehensionmonitoring skills. This investigation addressed this need by examining the
412
M. Fujiki and B. Brinton
ability of adults with mild to moderate levels of mental retardation to demonstrate comprehension-monitoring skills in a structured work task. Subjects were presented with trouble sources interspersed throughout a series of verbal directions from an employer. This particular setting was selected because of its importance for persons with mild and moderate levels of mental retardation who are working in community settings. Because it is necessary not only to detect but also to resolve these comprehension problems (e.g., by requesting clarification), the specific strategies used to signal comprehension problems were of particular interest. METHOD Subjects
Thirty subjects (13 men and 17 women) participated in the study. Nineteen subjects were sampled from community placements, and 11 were sampled from an institutional placement to provide a wide sampling of the population with mental retardation. Subjects were sampled from a larger group of persons with mental retardation to meet the following criteria: 1. Full scale Wechsler Adult Intelligence Scale-Revised (WAIS-R) (Wechsler, 1981) IQ within the mild to upper moderate range of mental retardation, The resulting group produced a mean WAIS-R full scale IQ of 61.43 (SD = 7.40). 2. Chronological age between 20 and 42 years. The resulting group produced a mean age of 28.9 years (SD = 6.27). 3. Auditory and visual status within normal limits as demonstrated by passing standard vision and hearing screening examinations. Normal visual status was defined as current binocular vision of 20/40 or better (with corrective lenses if needed). Normal auditory status was defined as current pure tone threshold of 35 dB or lower through the speech frequencies in the better ear. 4. Ambulatory, as demonstrated by the ability to walk independently. 5. Adequate verbal language skills to participate in conversation. All subjects produced a mean length of utterance, in morphemes, of over 4.0. Procedures
Each subject was assessed individually by the same examiner. Community-based subjects were seen at their supported work settings. Institutionalized subjects were seen in a research facility on the campus of the institution. All of these subjects were familiar with the research facility and had participated in previous studies. To motivate subjects to perform, the following procedure was used: Each subject initially met the investiga-
Comprehension Monitoring
413
tor in a situation structured as a job interview. During this 30-min interview, the investigator explained the task to be performed and told each subject that if they agreed to perform the required work they would be paid $3.30/h for their time (minimum wage at the time of the study). Although it was stressed that participation was voluntary, all subjects agreed to participate. Following the interview session, the trial phase of the study was initiated. Four containers were placed on a table in front of the subject. Each container was filled with colored paper clips, combs, rubber bands, or pencils. Each subject was asked to help the investigator fill gift bags with various combinations of these items. Subjects were initially presented with the following directions: I need to fill some gift bags. Each bag will have something different in it. I’d like you to help me find the things that go in each bag. I’ll tell you what I need and you give it to me, ok? Be sure to let me know if you don’t understand something.
Following these instructions, subjects performed four trial items, designed to insure that they were familiar with each of the objects and colors to be used in the task (e.g., Give me a red pencil, give me a green rubber band, etc.). If the subject performed all of the trial items appropriately, the experimental task was initiated. If any of the trial items was missed, further trial items were administered. It was necessary to administer additional trial items to only one of the 30 subjects who participated in the study. Another indication of the ability of the subjects to perform the direction-following task was the fact that 97% of the control directions (those that did not contain trouble sources) were completed appropriately on the first presentation. Control directions that were performed inappropriately were readministered a second time. The experimental task consisted of a set of 60 directions, 6 of which involved trouble sources (2 of each of 3 types of trouble sources), to represent 10% of the total number of directions. Directions containing trouble sources were randomly distributed. The three types of trouble sources used are described as follows: 1. Unintelligible (nonsense) word. The investigator substituted an unintelligible word in place of a key word (e.g., “Give me the blue /kliv/” and “Give me the green /gLIb/“). 2. Ambiguous referent. The investigator asked the subject for an item without providing enough detail to indicate the specific object desired (e.g., “Give me a rubber band” when there were several different colors of rubber bands available). 3. Compliance problem. The investigator requested that the subject act on an object that was not present (e.g., “Give me a red toothbrush” when no toothbrushes were present).
414
M. Fujiki and B. Brinton
The task took approximately 20 min to complete. The entire set of trouble source directions, examples of the control directions, and a more detailed explanation of administration procedures are available from the first author. Data Analysis All experimental procedures were videotape recorded. Each subject’s response to the directions involving trouble sources was coded into one of the four categories described here. Nonverbal behaviors, in addition to verbal behaviors, were used as indications of monitoring (Flavell et al., 1981; Skarakis-Doyle et al., 1990). The categories used reflect the view that comprehension monitoring must involve both evaluation and regulation (Ackerman, 1986; Baker, 1985). From this perspective, successful comprehension monitoring must involve both the detection of the comprehension problem (evaluation), and the attempt to resolve the problem through appropriate action (regulation). Specific categories of behavior were as follows: Effective comprehension monitoring 1. The subject immediately requested clarification of the direction, either verbally (e.g., “Huh?’ or “Could you repeat that?‘) or nonverbally (e.g., subject made a distinct gesture to the investigator as if to say, “What?‘). 2. The subject briefly (2 s or less) hesitated or glanced at the materials and then requested clarification. The subject did not visually scan (as indicated by looking from side to side) or physically search for the requested object. 3. The subject requested clarification in the process of reaching for an item (e.g., The subject reached for an item and at the same time said, “This?“). In these cases, the subject immediately reached for a single item without physically searching or shuffling through items. Request for clarification following attempt to comply. These responses also represented appropriate strategies for dealing with the comprehension problem but did not represent the same level of efficiency as the strategies discussed previously. “Effective monitoring” represented an immediate detection and subsequent action to resolve the trouble source. Responses categorized as “request for clarification following attempt to comply” represented detection of the trouble source only after the subject attempted to follow the direction and could not do so. (e.g., Revelle et al., 1985). 1. The subject physically touched or shuffled through the materials (longer than 2 s), then requested clarification. Because the 2-s delay was somewhat arbitrary, this criterion was always paired with other physically
Comprehension Monitoring
415
observable behaviors (e.g., physically searching through the materials) in categorizing the subject’s performance. 2. The subject visually scanned (by looking from side to side) the materials (longer than 2 s) and then requested clarification. 3. The subject reached for an item, stopped, and then requested clarification. Ineffective monitoring 1. The subject produced an immediate nonverbal reaction to the direction (e.g., suddenly tensing the body) but did not request clarification. This category also included cases in which a subject laughed upon hearing the direction. In all cases, the nonverbal response was followed by the subject’s handing the investigator an item, apparently guessing as to what was being requested. 2. The subject searched through the objects physically or scanned the objects visually for longer than 2 s, suggesting that there was some awareness that the direction could not be followed. However, rather than requesting clarification, the subject handed the investigator an item, again apparently guessing as to what was being requested. 3. The subject requested clarification of the wrong item in the sentence. For example, when asked for a blue /kliv/, a subject responded, “What color?’ No response to the trouble source 1. The subject handed the investigator an item without hesitation. The subject showed no response to the trouble source. Reliability Coding reliability was established using the following procedures. Following extensive training with the classification system, two investigators independently coded 20% of the data base using videotaped recordings of the data collection. The six subjects used to establish coding reliability were randomly selected from the sample. The percentage of agreement between investigators was calculated using the following formula: A/N x 100, where A was the total number of coding agreements, and N was the total number of observations to be coded. This procedure produced a percent agreement of 97%. Social Valia!ation In order to provide a measure of social validation, the same experimental procedure was also administered to a group of 23 adults. These individ-
416
A4. Fujiki and B. Brinton
uals were college students enrolled in a course in child language acquisition. Subjects participated on a volunteer basis and ranged in age from 18 to 34 years. These subjects appropriately monitored comprehension on 98% of the unintelligible items, 100% of the compliance problem items, and 85% of the ambiguous items. These results suggested that adults could be expected to identify the vast majority of trouble sources. However, it was of note that ambiguous trouble sources were the least powerful in eliciting a response. On several occasions (7 of 46), subjects responded to an ambiguous direction by simply handing the examiner an object. When debriefed and asked why they responded in this manner, these individuals noted that they thought that because no color was specified (e.g., “Give me a pencil”), the matter of color was their choice. RESULTS Statistical evaluation of the performance of the institutional and community subjects using unpaired t-tests on individual categories of performance indicated that group differences were not significant. Thus, the data from the groups were combined for further analyses. Table 1 presents the proportion of responses falling into each of the four basic response categories of (a) effective monitoring, (b) request for clarification following attempt to comply, (c) ineffective monitoring, and (d) no response to trouble source. For all three trouble sources, effective monitoring was the most frequently observed response. However, the proficiency with which subjects monitored trouble sources varied according to type of trouble source. As Table 1 illustrates, subjects identified and requested clarification of complianceproblem trouble sources at the highest rate of success, followed by unintelligible and ambiguous trouble sources.
TABLE 1 Response Patterns Used by Adults With Retardation
to Three Types of Trouble Sources Trouble Source
Effective monitoring Request for clarification following attempt to comply Ineffective monitoring No resuonse
Unintelligible
Ambiguous
Compliance
47% (28)
37% (22)
58% (35)
25% (15) 10% (6) 18% (11)
30% (18) 8% (5) 25% (15)
25% (15) 15% (9) 2% (1)
Note. Numbers in parentheses represent raw frequency totals.
417
Comprehension Monitoring
As a group, subjects identified compliance-problem trouble sources most readily. This observation was supported by analysis of the appropriate performance of the subjects. The two categories that represented appropriate performance were combined (“effective monitoring” and “request for clarification following attempt to comply”), and a one-way ANOVA with repeated measures was employed to determine if subject performance differed between trouble sources. Differences in performance between types of trouble sources were evident [F(2, 87) = 6.47, p < .Ol]. Subsequent post hoc analysis using the Newman-Keuls’ Test indicated that the ambiguous trouble sources were significantly more difficult than the compliance trouble sources (p c .Ol). Several different strategies of performance were included in the categories of “request for clarification following attempt to comply” and “ineffective monitoring.” The specific strategies grouped in each of these categories are presented in Tables 2 and 3, respectively. Table 2 presents specific strategies categorized as “request for clarification following attempt to comply.” These responses were considered as appropriate, but somewhat less efficient than responses categorized as “effective monitoring.” Subjects most frequently attempted to physically or visually search for the item, and then requested clarification of the direction. However, in a few cases, subjects reached for an item, stopped, and then requested clarification. It should be considered that although the observed searching behaviors were interpreted as attempts to comply with the direction, they may have been stalls for more processing time to deal with the problematic direction. Table 3 details responses categorized as “ineffective monitoring.” On 8-15% of the trials for the three types of trouble sources the subjects gave some nonverbal indication that they realized there was a problem, but then failed to request clarification. On most of these trials, the subTABLE 2 Strategies Classified as Request for Clarification
Following an Attempt to Comply
Unintelligible Overall percentage and frequency of occurrence Strategies Verbal clarification request following physical search or visual scan Verbal clarikation request following reach for item
Ambiguous
Compliance
25% (15)
30% (18)
25% (15)
22% (13)
27% (16)
25% (15)
3% (2)
3% (2)
0%
(0)
Note. Percentages represent proportion of all responses to each trouble source type. Numbers in parentheses represent raw frequency totals.
418
A4. Fujiki and B. Brinton
TABLE 3 to Represent “Ineffective Monitoring”
Strategies Considered
Overall percentageand frequency of occurfence Strategies Supplying item following search Nonverbal indication of awareness (no request for clarification) Requesting clarificationof wrong item
Unintelligible
Ambiguous
Compliance
10%(6)
8% (5)
15%(9)
5% (3)
0% (0)
8% (5)
5% (3) 0% (0)
1% (4) 2%a(1)
1% (4) 0% (0)
Note. Numbers in parenthesesrepresentraw frequency totals.
aPercentagesdo not sum to 8 because numbersare roundedto nearest whole.
jects eventually attempted to comply with the direction by handing the investigator an item. For example, one subject suddenly tensed upon hearing the direction but did not request clarification. Instead, she handed the investigator an item, apparently guessing as to what was wanted. Another subject consistently looked at the materials, looked up at the investigator, and then looked at the materials again without requesting clarification.
DISCUSSION
Subjects with mental retardation requested repair of the majority of trouble sources, either immediately or after an attempt to comply (74% overall). However, for over one-fourth of the inadequate directions, subjects either did not detect the trouble sources or did not provide feedback regarding the presence of the trouble sources to the investigator. In these cases, subjects appeared to guess at the requested item, either immediately (within 2 s) or following a scan or search for the item. In one case, a subject requested repair of an element that was intelligible while making no comment on the troublesome element (e.g. “What color?’ in response to “Give me a blue [kliv]“); however, most of the time that subjects did not respond to trouble sources, they attempted to follow the directions despite the trouble sources. It was clear that comprehension monitoring was influenced by the types of trouble sources embedded in the verbal directions. Subjects requested clarification of unintelligible directions and compliance-problem directions more often than they requested clarification of ambiguous directions. This result was not surprising considering the performance of the subjects on the social validation measure used and the performance of children on similar tasks as described in the literature (e.g., Revelle et al., 1985).
Comprehension
Monitoring
419
There are at least two possibilities to explain the particular difficulty some subjects experienced with ambiguous trouble sources. First, it is possible that for some individuals, the context of the task did not provide strong enough cues to demonstrate that both object and color were important in item selection. Subjects might then proceed as if the color of the object were optional. This was evidently the case for a few of the subjects without disabilities. However, it is also possible, in light of the subtlety of the trouble sources, that some subjects may not have detected the ambiguity. Regardless of the factors that influenced the subjects’ performance, the comprehension monitoring demonstrated in this task as a whole was less accurate than might be hoped considering the importance of the behaviors assessed to an individual’s functioning within a community. Most, if not all, supported and independent employment settings in which individuals with mental retardation work demand that employees follow verbal directions as accurately as possible. It is important to request clarification of unclear directions before completing the commands. Guessing at unclear elements will usually result in an error that detracts from the employee’s productivity. Although it can be argued that the subjects in this study detected more trouble sources than they signaled verbally, in a practical sense th,s lnade little difference in how efficiently they would function in a work setting. The findings of this study suggest several clinical implications. First, requesting repair of genuine misunderstandings in work tasks is an important skill that may warrant attention in intervention. It is likely that some individuals with mental retardation may have difficulty detecting and resolving comprehension problems beyond that which might be expected considering their general levels of functioning. Treatment programs designed to prepare and support individuals with mental retardation in securing and maintaining employment in community settings might include strategies for handling miscommunications. Requesting clarification of trouble sources might be facilitated in the context of improving work productivity and accuracy. Specifically, clinical treatment programs could evaluate and subsequently treat (if indicated) recognition of instances where comprehension of important information is jeopardized. With regard to recognizing ambiguity in verbal instructions, detection of subtle trouble sources could be stressed by comparing “What do I know about this task?’ with “What do I need to know about this task?‘. Strategies for requesting clarification in efficient, socially acceptable ways could be suggested. Although immediate recognition and request for repair constitutes the most efficient manner of resolving miscommunications, requesting repair after a failed attempt to complete a task could also be a viable method. Finally, acknowledging and acting upon repairs elicited could constitute the final step in treatment.
420
A4. Fujiki and B. Brinton
Repairing trouble sources in interaction is an interdependent behavior negotiated between conversational partners. Even though the behavior itself is interdependent, the ability to initiate a repair sequence is a skill that facilitates social independence. Knowing how and when to request help can widen the social sphere in which one can function. In conclusion, it should be noted that despite the ecological validity of the task, it is not clear whether the performance observed in these subjects was representative of behavior in employment settings. Further research in actual employment settings is needed to clarify this issue. In addition, it is not clear whether subjects failed to request clarification because of social reasons (e.g., fear of questioning the investigator) or whether they actually were unaware of the trouble source. Both of these issues should be explored in further investigations. REFERENCES Abbeduto, L., Furman, L., & Davies, B. (1989). The relationship between receptive language and mental age in persons with mental retardation. American Journal on Mental Retardation, 93, 535-543.
Ackerman, B. P (1986). Children’s sensitivity to comprehension failure in interpreting a nonliteral use of an utterance. Child Development, 57,485-497. Baker, L. (1985). How do we know when we don’t understand? Standards for evaluating text comprehension. In D. Forrest-Pressley, G. E. MacKinnon, & G. T. Wailer (Eds.), Metacognition, cognition and human performance (Vol. I. pp. 155-205). Orlando, FL: Academic Press. Bonitatibus, G. (1988). Comprehension monitoring and the apprehension of literal meaning. Child Development, 59,60-70.
Brooks, I? H., Sperber, R., &t McCauley, C. (1984). Learning and cognition in the mentally retarded. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Chapman, D. L., & Nation, J. E. (1981). Patterns of language performance in educable mentally retarded children. Journal of Communication Disorders, 14.245-254. Dollaghan, C. (1987). Comprehension monitoring in normal and language-impaired children. Topics in Language Disorders, 7,45-60.
Dollaghan, C., & Kaston, N. (1986). A comprehension monitoring program for languageimpaired children. Journal of Speech and Hearing Disorders, 51,264-271. Ezell, H. K., & Goldstein, H. (1991). Observational learning of comprehension monitoring skills in children who exhibit mental retardation. Journal of Speech and Hearing Research, 34. 141-154.
Flavell, J. (1981). Cognitive monitoring. In W. P. Dickson (Ed.), Children’s oral communication skills (pp. 35-60). New York Academic Press. Flavell, J., Speer, J., Green, F., & August, D. (1981). The development of comprehension monitoring and knowledge about communication. Monographs of the Society for Research in Child Development, No. 192.
Markman, E. (1977). Realizing that you don’t understand: A preliminary investigation. Child Development, 48,986-992.
Markman, E. (1979). Realizing that you don’t understand: Elementary school children’s awareness of inconsistencies. Child Development, 50.643-655. Markman, E. (1981). Comprehension monitoring. In W. P. Dickson (Ed.), Children’s oral cornmunication skills (pp. 61-84). New York: Academic Press.
Comprehension Monitoring
421
Revelle, G. L., Welhnan, H. M., & Karabenick, J. D. (1985). Comprehension monitoring in preschool children. Child Developmen?, 56,654-663. Skarakis-Doyle, E., MacLellan, N., & Mullin, K. (1990). Nonverbal indicants of comprehension monitoring in language-disordered children. Journal of Speech and Hearing Disorders, 55, 461-467.
Skarakis-Doyle, E., & Mullin, K. (1990). Comprehension monitoring in language-disordered children: A preliminary investigation of cognitive and linguistic factors. Journal of Speech and Hearing Disorders, 55,700-705.
Wechsler, D. (1981). Wechsler Adult Intelligence Scale-Revised. Corporation.
Cleveland: Psychological