The convergent validity of the Matson Evaluation of Social Skills for Individuals with Severe Retardation (MESSIER)

Pergamon

Research in Developmental Disabilities, Vol. 19, No. 6, pp. 493–500, 1998 Copyright © 1998 Elsevier Science Ltd Printed in the USA. All rights reserved 0891-4222/98 $19.00 1 .00

PII S0891-4222(98)00020-1

The Convergent Validity of the Matson Evaluation of Social Skills for Individuals with Severe Retardation (MESSIER) Johnny L. Matson Louisiana State University

Carla Beth Carlisle Northwestern State University

Jay W. Bamburg Louisiana State University

The convergent validity of the Matson Evaluation of Social Skills for Persons with Severe Retardation (MESSIER) was examined. We administered the MESSIER and the Vineland Adaptive Behavior Scales (VABS) to 892 individuals with severe and profound mental retardation residing in a residential state facility. The MESSIER subscales were compared to equivalent subdomains from the VABS that have been demonstrated through research to be both reliable and valid. Significant positive correlations were found between corresponding MESSIER subscales and VABS subdomains on social behaviors. The findings also indicated that this pattern of positive correlation remained constant when comparing both verbal and nonverbal social behaviors. The implications and clinical utility of the findings are discussed. © 1998 Elsevier Science Ltd

Address correspondence to Johnny L. Matson, Ph.D., Department of Psychology, Audubon Hall, Louisiana State University, Baton Rouge, LA 70803-5501.

493

494

Matson et al.

One of the greatest challenges in the field of mental retardation has been to habilitate persons to their maximum level of independence (Matson, Fee, Coe, & Smith, 1991). A method for increasing independence is through training social and adaptive skills. The assessment and training of social and adaptive skills is important for a number of reasons. First, social and adaptive skills deficits can compromise successful transition from institutional to community living (Jacobson & Schwartz, 1991; Doll, 1953). Second, deficits in these areas may contribute to the etiology of psychiatric disorders and behavioral problems (Borthwick-Duffy & Eyman, 1990; Matson & Sevin, 1994). Finally, social and adaptive deficits often persist in living areas where the philosophy of care consists of passive learning rather than active treatment (Matson & Hammer, 1996). Consequently, it is imperative that professionals become active in the assessment and training of social skills in individuals with mental retardation. Though definitions have varied, social and adaptive skills are typically defined as observable and measurable interpersonal behaviors that promote independence, social acceptability, and quality of life (McFall, 1982; Bellack, 1983). The two most commonly used social skills assessment techniques include observation and rating scales (Bellack, 1979). Observation techniques are conducted in either naturalistic settings or analog conditions. Naturalistic observations usually occur in settings that the individual frequents and include observations by several raters based on target behaviors identified before observations (Marchetti & Campbell, 1990). Analog assessments occur in artificial environments and involve the manipulation of situational variables to evaluate a person’s overall repertoire of social behaviors (Castles & Glass, 1986). Although these techniques are viable in some situations, variables such as differences in conversational skills, pre-established operations, and situational specificity often limit their reliability, generalizability, and predictive abilities (VanHasselt, Hersen, & Bellack, 1981). Behavior checklist and rating scales are the most frequently used tools to assess social skills in individuals with severe or profound mental retardation. Rating scales have proven to be more objective, reliable, and efficient than other frequently used methods for assessing social skills (Marchetti & Campbell, 1990). Further, one can obtain a wide range of information from a variety of informants with little difficulty. The disadvantages of some social skills rating scales is that it is difficult to control the many factors that can influence the accuracy of ratings. Therefore, low interrater and test-retest reliability have often confounded the use of such scales with persons with severe or profound mental retardation (Marchetti & Campbell, 1990). However, two scales for rating social and adaptive behaviors have avoided problems with psychometrics by recommending use as part of a multimethod assessment. These two scales are the Vineland Adaptive Behavior Scales (VABS) (Sparrow, Balla, & Cicchetti, 1984) and the Matson Evaluation of Social Skills for Persons with Severe Retardation (MESSIER) (Matson, 1995).

Validity of Social Skills Measure

495

The VABS (Sparrow et al., 1984) is a well validated measure of adaptive behaviors in persons with severe and profound mental retardation. It has demonstrated adequate psychometric properties across the 4 domains, 11 subdomains, and the adaptive behavior composite contained in the scale. The VABS has been used extensively in the field of mental retardation to establish the overall level of adaptive functioning. However, because of the general nature of the scale, certain areas such as social skills are only generally surveyed. Specifically, the VABS contains only a limited number of items pertaining to social skills in the Socialization Domain and reflects only social abilities while disregarding social deficits or inappropriate behavioral excesses. Therefore, a scale that is specific to the assessment of social skills in persons with severe and profound mental retardation appears warranted. One scale that addresses this need is the MESSIER (Matson, 1995). The MESSIER is an 85-item questionnaire designed to globally assess social skills in severely and profoundly handicapped individuals. It consists of six subscales including: 1) positive verbal skills; 2) positive nonverbal skills; 3) general positive skills; 4) negative verbal skills; 5) negative nonverbal skills; and 6) general negative skills. The MESSIER has demonstrated good reliability, as internal consistency was high for the scale as a whole as well as for each clinically derived subscale (r 5 0.76 to 0.96). Test-retest and interrater reliability were adequate as well (r 5 0.86, r 5 0.79 respectively) (Matson, LeBlanc, Weinheimer, & Cherry, in press). Factor analysis conducted on the MESSIER yielded two significant factors, positive and negative social skills (Paclawskyj, Rush, Matson, & Cherry, 1998). An initial analysis of the convergent validity of the MESSIER was evaluated by determining the correlation of the MESSIER to sociometric rankings, although the outcome of this analysis was variable (r 5 0.09 to 0.94) (LeBlanc, Matson, Cherry, & Bamburg, 1998). Therefore, further studies of the validity of the MESSIER appear warranted. The purpose of the present research was to extend the psychometric properties of the MESSIER by further examining convergent validity. Specifically, we were interested in the extent to which the positive subscales from the MESSIER and the total MESSIER score would positively correlate with corresponding subdomains on social behavior from the VABS. Statistically significant correlations between an instrument of interest and a well-established related instrument help provide support for the convergent validity of the new instrument (Anastasi, 1988). The initial attempt to validate the MESSIER yielded mixed results, but the research attempted to correlate MESSIER scores with highly subjective sociometric rankings (LeBlanc et al., 1998). As the domains and subdomains of the VABS have consistently proven reliable and valid in past research, correlating the MESSIER with corresponding VABS subdomains would be significant in establishing the validity and utility of the social skills rating scale.

496

Matson et al.

METHOD Participants Participants included 892 residents of a state facility for persons with mental retardation in central Louisiana. Approximately 16% were diagnosed with severe mental retardation and 84% with profound mental retardation as determined by scores on a standardized measure of intelligence and the VABS. The participants ranged in age from 5 to 88 (mean age 5 42 years, standard deviation (SD) 5 15.1). Participants were predominantly Caucasian (72%); all others were African-American (28%). In addition, 62% of the sample were male, whereas 38% were female.

Measures The MESSIER is an 85-item questionnaire designed to assess social strengths and weaknesses in persons with severe and profound mental retardation (Matson, 1995). The items were developed from the clinical experience working in the field of developmental disabilities. These items were derived from lists formulated by professionals that included areas of social skills that were normally the focus of training as well as a survey of common assessment instruments used in the population. The MESSIER is administered by a trained examiner to a caregiver who has worked with the client for at least 6 months. The items of the MESSIER are grouped into six clinically derived subscales: 1) positive verbal; 2) positive nonverbal; 3) positive general; 4) negative verbal; 5) negative nonverbal; and 6) general negative. Each item is rated on frequency using a four-point Likert: never (0), rarely (1), sometimes (2), and often (3). Items endorsed are transcribed onto a scoring profile under their respective subscales. This method of scoring allows the clinician to examine which subscale(s) have the most items endorsed. Recent studies have indicated sound psychometric properties on the MESSIER (LeBlanc et al., in press). The VABS was used as a measure of adaptive skills in this study. It consists of 4 domains, 11 subdomains, an Adaptive Behavior Composite, and a maladaptive behavior domain. Reliability coefficients for internal consistency, interrater, and test-retest estimates are high averaging in the 0.80s and 0.90s. The socialization domain, which is the domain of primary interest in this study, contains 58 items and has proven to be very reliable and valid. There are three subdomains of the socialization domain: 1) interpersonal relationships; 2) play and leisure time; and 3) coping skills. Each item can receive a score of 2 (yes, usually), 1 (sometimes or partially), 0 (no, never), N (no opportunity), or DK (don’t know). The questions are general questions about social skills and are listed in the order they should be developmentally achieved. Test of internal consistency on this domain resulted in coefficients above 0.80, whereas measures of construct validity resulted in coefficients above 0.70.

Validity of Social Skills Measure

497

TABLE 1 Means and Standard Deviations of MESSIER Subscales and VABS Subdomains

MESSIER General positive Positive verbal Positive nonverbal Total score VABS Interpersonal relationships Coping skills Play and leisure time Socialization domain score

Mean

SD

34.42 9.80 24.21 66.77

18.53 12.01 10.71 29.93

17.55 3.68 8.47 29.30

10.62 8.84 10.08 22.76

Administration Direct care staff who had worked with the clients for at least 6 months were used as the informants for the administration of the VABS and the MESSIER. Master’s-level psychologists, qualified mental retardation professionals, and graduate students in psychology administered the VABS and the MESSIER to direct care staff. Each examiner received comprehensive training on the administration of the MESSIER and the VABS. Administration of the two instruments took place in a quiet place in the clients’ living area and lasted approximately 45 min.

RESULTS Before conducting the principle statistical analyses, we established the means and SDs for the MESSIER subscales and VABS subdomains of interest. Means and standard deviations for the positive verbal, positive nonverbal, general positive, and total positive score were established for the MESSIER and the interpersonal relationships, coping skills, play and leisure time, and socialization domains for the VABS. Means and SDs can be found in Table 1. Next, we established correlations (Pearson Product-Moment) for MESSIER subscale scores and the VABS domain or subdomain with which they seemed most closely related based on item examination. First, we correlated the Interpersonal Relationships subdomain on the VABS with the General positive subscale of the MESSIER. The calculated statistic indicated a significant positive correlation (r 5 0.84, p , 0.01). Next, we correlated the Coping Skills subdomain of the VABS with the Positive verbal subscale from the MESSIER. A significant positive correlation (r 5 0.52, p , 0.01) was found in this analysis. Third, the Play and Leisure Time subdomain on the VABS was correlated with the Positive nonverbal subscale on the MESSIER. The calculated statistic indicated as significant positive correlation in these areas (r 5

498

Matson et al.

TABLE 2 Correlations Between VABS Subdomains and MESSIER Subscales

MESSIER subscales Genpos Posver Posnon Mestot

Interper

Coping

Playleis

Soc

0.84* — — —

— 0.52* — —

— — 0.71* —

— — — 0.77*

*p , 0.01.

0.71, p , 0.01). Finally, we correlated the Socialization Domain from the VABS with the Total positive score from the MESSIER. This analysis also yielded a significant positive correlation (r 5 0.77, p , 0.01). A listing of each of these correlations can be found in Table 2.

DISCUSSION Social skills are an important factor influencing overall functioning (Guralnick, 1986). Low social skills have been consistently correlated with severe behavior problems, psychiatric illness, and unsuccessful community placement in persons with mental retardation (Chadsey-Rusch, 1992). Very little research exists on the social skills of persons evincing severe or profound mental retardation, which is probably a result of the lack of adequate measures to measure social skills in this population. The MESSIER could possibly aid researchers to learn more about social skills in this population by providing a standardized scale for measuring the constructs. One of the defining criteria for mental retardation is deficits in adaptive functioning, and level of social skills is a more specific component of adaptive functioning (APA, 1994; AAMR, 1992). To prepare persons with mental retardation for independent living, an increase in adaptive and social functioning must be achieved. Although intelligence level cannot usually be increased, level of adaptive functioning normally can through social skills training (Hazinski & Matson, 1985). Another means of achieving more independent living is to decrease maladaptive behaviors and psychiatric symptomology, which can also be achieved by increasing social and adaptive behaviors (Helsel & Matson, 1988; Matson & Barrett, 1993). Therefore, social skills training in essential in increasing overall adaptive functioning and decreasing maladaptive behaviors and should always be included in treatment packages for persons with mental retardation. The MESSIER could provide professionals working with individuals with severe and profound mental retardation with a social skills assessment tool that is practical, economical, and essential for identifying social areas that should be the focus of training in treatment packages.

Validity of Social Skills Measure

499

The MESSIER has several strengths over the VABS in measuring social skills in persons evincing severe or profound mental retardation. First, the MESSIER has a larger number of social-skills-specific questions than the VABS, which is a more global rating of adaptive functioning across a larger number of developmental and adaptive areas. The specificity of the MESSIER is important because it depicts social skills deficits that should be core ingredients of individualized treatment packages. Additionally, the MESSIER provides a profile of positive social skills, negative social skills, and maladaptive behavior excesses, whereas the VABS is limited to providing information about social abilities only. It is important for professionals to recognize an individual’s positive and negative social attributes rather than concentrating on positive social skills possessed by an individual when developing treatment and habilitation plans. The initial psychometric properties of the MESSIER have been well established through reliability studies (Matson et al., in press). These results, combined with the results of the current validity study, help to establish the MESSIER as a reliable and valid behavioral checklist for assessing social skills in persons with severe or profound mental retardation. The AAMR recently issued a call for valid and reliable social skills assessments to aid in the new definition of mental retardation. As the MESSIER has been shown to be a reliable and valid instrument for assessing social skills in persons with severe and profound mental retardation, it is possible that the scale can fulfill this need by providing a functional and economical alternative to evaluating social skills in this population.

REFERENCES American Association on Mental Retardation. (1992). Mental retardation: Definition, classification, and systems of supports (9th ed.). Washington, D.C.: AAMR. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, D.C: APA. Anastasi, A. (1988). Psychological testing (6th ed.). New York: Macmillan Publishing Company. Bellack, A. S. (1979). A critical appraisal of strategies for assessing social skill. Behavioral Assessment, 1, 157–176. Bellack, A. S. (1983). Recurrent problems in the behavioral assessment of social skills. Behaviour Research and Therapy, 21, 29 – 41. Borthwick-Duffy, S. A., & Eyman, R. K. (1990). Who are the dually diagnosed? American Journal on Mental Retardation, 94, 586 –595. Castles, E., & Glass, C. (1986). Training in social and interpersonal problems solving skills for mildly and moderately retarded adults. American Journal of Mental Deficiency, 91, 35– 42. Chadsey-Rusch, J. (1992). Toward defining and measuring social skills in employment settings. American Journal on Mental Retardation, 96, 405– 418. Doll, E. A. (1953). The measurement of social competence. Minneapolis: Education Publishers. Guralnick, M. J. (1986). Children’s social behavior: Development, assessment, and modification. Orlando, FL: Academic Press. Hazinski, L., & Matson, J. L. (1985). A comparison of social skills training and direct reinforcement methods with mentally retarded adults. Scandinavian Journal of Behavior Therapy, 14, 23–31.

500

Matson et al.

Helsel, W. J., & Matson, J. L. (1988). The relationship of depression to social skills and intellectual functioning in mentally retarded adults. Journal of Mental Deficiency Research, 32, 411– 418. Jacobson, J. W., & Schwartz, A. A. (1991). Evaluating living situations of people with developmental disabilities. In J.L. Matson & J.A. Mulick (Eds.). Handbook of mental retardation (2nd ed., pp.35– 62). New York: Plenum Press. LeBlanc, L. A., Matson, J. L., Cherry, K. E., & Bamburg, J. W. (1998). An examination of the convergent validity of the MESSIER with sociometric ranking. Manuscript in progress. Louisiana State University. Marchetti, F. M., & Campbell, A. J. (1990). Social Skills. In J.L. Matson (Ed.). Handbook of Behavior Modification with the Mentally Retarded (2nd ed., pp. 333–355). New York: Plenum Press. Matson, J. L. (1995). The Matson Evaluation of Social Skills for Individuals with Severe Retardation. Baton Rouge, LA: Scientific Publishers, Inc. Matson, J. L., & Barrett, R. P. (Eds.) (1993). Psychopathology in the Mentally Retarded (2nd ed.). Boston: Allyn and Bacon. Matson, J. L., Fee, V. E., Coe, D. A., & Smith, D. (1991). A social skills program for developmentally delayed preschoolers. Journal of Clinical Child Psychology 20, 428 – 433. Matson, J. L., & Hammer, D. (1996). Assessment of social functioning. In J.W. Jacobson & J.A.Mulick (Eds.). Manual of diagnosis and professional practice in mental retardation (pp. 157–164). Washington, DC: American Psychological Association. Matson, J. L., LeBlanc, L. A., Weinheimer, B., & Cherry, K. E. (In press) Reliability of the Matson Evaluation of Social Skills for Individuals with Severe Retardation. Behavior Modification. Matson, J. L., & Sevin, J. A. (1994). Theories of dual diagnosis in mental retardation. Journal of Consulting and Clinical Psychology, 62, 6 –16. McFall, R. M. (1982). A review and reformulation of the concept of social skills. Behavioral Assessment, 4, 1–33. Paclawskyj, T. R., Rush, K. S., Matson, J. L., & Cherry, K. E. (1998). Factor structure of the Matson Evaluation of Social Skills for Individuals with Severe Retardation (MESSIER). Manuscript in progress. Louisiana State University. Sparrow, S. S., Balla, D. A., & Cicchetti, D. V. (1984). Vineland Adaptive Behavior Scales: Interview Edition Survey Form Manual. Circle Pines, MN: American Guidance Service. VanHasselt, B., Hersen, M., & Bellack, A. E. (1981). The validity of role play tests for assessing social skills in children. Behavior Therapy, 12, 202–216.