- Email: [email protected]
The validity of role play tests for assessing social skills in children
BEHAVIOR THERAPY 12, 202-216 (1981)
The Validity of Role Play Tests for Assessing Social Skills in Children VINCENT B . VAN HASSELT
University of Pittsburgh MICHEL HERSEN
Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine ALAN S. BELLACK
University of Pittsburgh
A correlational study was conducted in an effort to examine the validity and reliability of role play tests for assessing social skills in children. Twenty-two male and twenty female elementary school children, ranging in age from 8-10 years, responded to role play scenarios involving social interactions. In addition to the role play procedure, a number of other assessment strategies were utilized: (1) observations of behavior in naturalistic settings, (2) sociometric ratings, and (3) teachers' ratings. Results did not support the validity of the role play test. Correlations between role play performance and criterion measures were generally quite low. Test-retest reliability of the role play tests also was unacceptable. Further, results indicated that while several significant relationships were found between children's sociometric scores and teachers' ratings of interpersonal behavior, little association was found between these measures and global ratings of friendliness in role play and in vivo situations. Results are discussed in terms of
This article is based on the senior author's Masters Thesis, conducted under the direction of the second and third authors, and submitted to the Graduate Faculty of Arts and Sciences, University of Pittsburgh, in partial fulfillment of the requirements for the Master of Science degree. Portions of this paper were presented at the 87th Annual Convention of the American Psychological Association, New York, NY, September, 1979. The authors thank Denise A. Herbol, Curtis Gibson, Steven Stern, Denise Reid, Ronald Barbati, and Carla Costanza for their varied contributions throughout this project and Susan Giusti, Cheryl Kubelick, Bill McDonald, and Tim Rusnak of the Falk Elementary School for their extensive cooperation. Requests for reprints should be sent to Michel Hersen, Department of Psychiatry, Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine, Pittsburgh, PA 15261. 202 0005-7894/81/0202-021651.00/0 Copyright 1981 by Association for Advancement of Behavior Therapy All rights of reproduction in any form reserved.
VALIDITY OF ROLE PLAY TESTS
203
the validity and utility of role play tests for children, the relationship of these data to previous research ~n social skills assessment, and the heuristic value of examining research concerning children's interpersonal functioning from developmental and social psychological perspectives.
Increased attention has been directed to the assessment and remediation of interpersonal skill deficiencies in children (see reviews by Beck & Forehand, Note 1; Combs & Slaby, 1977; Michelson & Wood, in press; Van Hasselt, Hersen, Whitehill, & Bellack, 1979). This growing interest is attributed to a large body of research demonstrating an association between level of children's social functioning and their long-term adjustment (cf. Cowen, Pederson, Babigian, Izzo, & Trost, 1973; Kagan & Moss, 1962; Roff, 1961). These findings underscore the need to develop effective interventions for children who demonstrate poor social competence (i.e., aggression, unassertiveness, social withdrawal). However, to evaluate adequately the efficacy of treatment regimens applied to this population, employment of valid and reliable assessment strategies is essential. While behavior therapists generally have favored in vivo assessment methods (Goldfried & Kent, 1972; Mischel, 1972), issues of ethics, economics, and pragmatics often preclude use of such strategies. Thus, analogue strategies (e.g., behavioral avoidance tasks, role play tests) have frequently been used as viable alternatives. For example, in the area of social skills research, role play tests "have become the hallmark o f . . . assessment" (Bellack, 1979). Regardless of how appealing use of such procedures may be (they are convenient, economical, face valid), their value is questionable if they are psychometrically unsound. Recently, Bellack and his colleagues (Bellack, Hersen, & Lamparski, 1979; Bellack, Hersen, & Turner, 1978, 1979) found that role play procedures were n o t externally valid for chronic psychiatric patients and college students (i.e., little correspondence was found between role play performances and interpersonal behavior in more naturalistic settings). Reardon, Hersen, Bellack, and Foley (1979) presented the first data concerning the relationship between children's role played assertive behavior and their responses in other social performance tasks. Results indicated that children classified as high assertive or low assertive on the basis of overall ratings differed significantly on several behavioral response components (e.g., response latency, speech duration). However, behavioral ratings of overall assertiveness showed little relationship to self-report or teacher ratings of assertiveness. While the study by Reardon et al. (1979) provides important preliminary data pertaining to children's interpersonal behavior, the external validity of their primary assessment tool (i.e., role play test) has yet to be determined. The primary purpose of our study is to determine the external validity of role play tests for assessing social skills in children. Correspondence between response components in children's role play
204
VAN H A S S E L T , H E R S E N , A N D B E L L A C K
tests and in more naturalistic situations is examined. A second purpose of this investigation is to examine the relationship between behavioral assessment and sociometric status of children.
METHOD Subjects Twenty-two males and twenty females between the ages of 8 and 10 were selected at random from class rosters of the Falk School of the University of Pittsburgh. The male group included six third-graders, eight fourth-graders, and eight fifth-graders. The female group included four third-graders, eight fourth-graders, and eight fifth-graders. The children were informed of the voluntary status of their participation. Consent was obtained via form letters sent to parents which described procedures and objectives of the project. Procedure The procedure involved observation of children's interpersonal behavior in four different settings: (1) problem solving groups, (2) standardized interviews, (3) role play tests, and (4) dyadic interactions. Sociometric instruments and teachers' rating scales also were administered. All subjects were first involved in problem solving groups which were conducted at the Falk School. Standardized interviews (also administered at Falk) as well as role play tests and dyadic interactions (both conducted at the Clinical Psychology Center of the University of Pittsburgh) were carried out approximately 1 week after problem solving sessions. Dyadic interactions immediately preceded administration of role play tests. Onehalf the subjects participated in the dyadic interactions and role play tests first, while the other half received the standardized interview first. Administration of the remainder of the tasks was scheduled according to the availability of subjects and/or teachers, with the restriction that no two tasks could be administered on the same day.Role play tests. Two forms of role play tests for children were employed in the present investigation: (1) the Children's Interpersonal Behavior Test (CIBT) and (2) the Conversation Probe (CONPROBE) (Whitehill, Hersen, & Bellack, 1980). 1 (1) CIBT. The CIBT is a modified version of the Behavioral Assertiveness Test for Boys (BAT-B) (Reardon et al., 1979) and the Behavioral Assertiveness Test for Children (BAT-C) (Bornstein, Bellack, & Hersen, 1977). It consists of six role play situations designed to be typical of social situations children are likely to encounter in their natural environments. Teacher input and feedback regarding realism and representativeness of scene content were obtained during item construction. Half the scenes in 1Copies of the CIBT and CONPROBE are available upon request from Vincent B. Van Hasselt, Clinical PsychologyCenter, University of Pittsburgh, Pittsburgh, PA 15260.
VALIDITY
OF ROLE PLAY TESTS
205
the CIBT involved a male role play partner while the other half involved a female role play partner. Unlike the Behavioral Assertiveness Test--Revised (BAT-R) (Eisler, Hersen, Miller, & Blanchard, 1975) or current behavioral assertiveness tests for children (in which termination of the subject's response to the prompt serves as a cue for the experimenter to present the next situation), the role play partner in the CIBT followed the subject's initial response with two additional prearranged counterresponses to allow for an extended interchange. This format was designed to make CIBT items more similar to real-life encounters, which often are longer than interactions possible with one-line prompts and responses on instruments such as the BAT-R (Curran, 1978). (2) CONPROBE. To assess extended interactions necessary for initiating a conversation Whitehill et al. (1980) developed the CONPROBE. This assessment device consists of single interpersonal situations similar in format to those found in the CIBT. However, the CONPROBE enables the subject to engage in prolonged (1-rnin) conversations with the role model. Also, with the CONPROBE, the confederate does not deliver a prompt. Rather, the child is asked to initiate and maintain an interaction following description of the interpersonal situation. Confederates were directed to respond to subjects in a facilitating, nondominating manner, and were asked to break silences of longer than 5 sec by introduction of a brief, facilitative question. Procedures for administration of CIBT and CONPROBE were similar to those described in previous social skills investigations with children (see Bornstein, Bellack, & Hersen, 1977, 1980; Reardon et al., 1979). Role play partners (one male and one female) for both the CIBT and CONPROBE were undergraduate psychology majors who received approximately 8 hours of training in these procedures. The same set of role play partners was used for both tasks. A male graduate student in clinical psychology served as narrator for administration of the role play test. Two additional undergraduates were involved in the rating of behavioral components, while another two were involved in independent ratings of overall friendliness on role plays. Scoring. Videotaped responses to CONPROBE and CIBT were subsequently rated on behavioral components of social skill identified by Eisler and his colleagues (Eisler, Hersen, & Miller, 1973; Eisler et al., 1975) and Whitehill et al. (1980). Overall friendliness also was rated on these tasks. Several other assessment instruments were employed in this investigation and are described in the following sections. Dyadic interactions (DYINT). Dyads consisting of two experimental subjects were observed in a waiting room situation. Twenty dyads included one male and one female subject. One dyad consisted of two males. When subjects scheduled for a particular session arrived at the Clinical Psychology Center, they were led to a room in groups of two and informed that their task (the role play test) would be administered in a
206
VAN H A S S E L T , H E R S E N , A N D B E L L A C K
few minutes. They were further asked to stay seated and told that they were free to converse until it was time to begin. The observation room was empty except for a pair of chairs about 5 ft apart, facing each other at an angle of about 90°. Such positioning was suggested by Rosenfeld (1965) to constitute a "neutral affiliative distance." It has the advantage of facilitating observation of nonverbal behaviors. Dyadic interactions were of 5-min duration. They were videotaped through a one-way mirror and rated for eye contact, speech duration, intonation, voice volume, speech disturbances, smiles, and overall friendliness. Standardized interview (SI). Each subject was individually interviewed by a female undergraduate research assistant who was not involved in administration of the other procedures. She was trained to administer this task in a uniform manner. Subjects in the present study were asked six open-ended questions (e.g., How are you today? How do you like school this year?). Interview sessions were audiotaped and subsequently rated for response latency (to questions), speech duration, intonation, voice volume and overall friendliness. Problem solving group (PSG). Subjects were observed participating in an open-ended interpersonal interaction which required them to analyze and produce a solution for an interpersonal dilemma. The problem, adapted from Shaftel and Shaftel (1967), presented children with a situation in which they had to decide which of three friends is to pay for a lamp broken when the youngest of the three hits it with his frisbee. This occurred after the child's mother told the older children to play frisbee outdoors. Groups consisted of three children randomly selected from the pool of experimental subjects. The experimenter first described the dilemma to the group. Following description of the problem, subjects were asked (as a group) a series of questions: (a) What is the problem here? (b) What are all the possible ways to solve this problem? (c) What might happen if each of these solutions is used? (d) Which solution is best? A drawing consisting of all the characters in the problem was provided to the group for reference purposes. Group interactions were 16 min in duration and were observed by trained raters. An interval recording procedure was utilized in which each child was alternately observed in 2-min blocks (eight 15-sec intervals) by two raters for occurrence or nonoccurrence of the following response components: smiles, interruptions, and range of interactions. Observers were undergraduates whose training continued until a criterion of 80% inter-rater agreement was reached. Each of the two raters was assigned to observe two children in a session. In all cases, there was overlap of one child in behavioral observations which served as a reliability check. Thus, reliability ratings were based on observations of approximately 33% of the sample. Sociometrics. The sociometric assessment method used in this study was adapted from Hymel and Asher (Note 2). (1) Positive peer nomination sociometric. Each child was given a list of all his/her classmates and asked to circle the names of three children
VALIDITY OF ROLE PLAY TESTS
207
they "especially like in s c h o o l . " Children were provided with a class roster to avoid the possibility that a child was not nominated because he/ she was forgotten. (2) Negative peer nomination sociometric. Each child was given a list of his/her classmates and asked to circle the names of three children he/she "especially did not like in school." Class rosters were again provided to students to minimize omissions. Subjects' scores on the above measures consisted of the number o f nominations received from same-sex peers. Only same-sex nominations were used since previous research has shown that children of this age group typically give low ratings to opposite-sex peers (Criswell, 1939; Singleton & Asher, Note 3). (3) Roster and rating scale sociometric. Children were asked to give ratings of each classmate on a 1 to 5 scale in answer to the question, " H o w much do you like to play with this person at s c h o o l ? " A rating of 1 indicated that the child did not like to play with that classmate. Children were provided with an alphabetized list of their classmates on which each name was followed by the numbers 1, 2, 3, 4, and 5. A child's score on the play ratings consisted of the average rating received from same-sex peers.
Teacher's checklist of interpersonal behavior (IBT) (Keller, Deysach, Hiers, Ross, & Moore, 1976). The IBT is a measure of interpersonal functioning in children. This instrument includes 17 items which can be adequately assessed within the school context. Some of these items include: "fights with other children," "complies with teacher's r e q u e s t s , " and "participates well in group activities." An additional item, used by Marsh, Serafica, and Barenboim (Note 4) included in the present study was "overall level of interpersonal skill relative to other children of the same a g e . " Classroom teachers rated each subject on each of the first 16 items using the following six categories: almost always, very often, often, sometimes, rarely, and never. In addition, subjects were rated with reference to the final item (overall interpersonal effectiveness) as follows: top 5%, top 25%, middle 25%, lowest 25%, and lowest 5%. The first 16 items were scored 1 to 6 (never to almost always) to represent the six possible categories (lowest 5% to top 5%). Thus, there were 17 individual scores for each subject. Physical attractiveness ratings (PA). Four additional undergraduates (three females, one male) independently rated each subject on physical attractiveness. Each judge observed a 30-sec video segment (no audio) from the middle of an individual subject's role play and employed a 5point scale (from 1 = very unattractive to 5 = very attractive).
RESULTS Reliability of Ratings Reliability (agreement) for ratings of each measure was assessed by comparing the scores of a primary rater with a reliability rater for 25%
208
VAN HASSELT,
HERSEN,
AND BELLACK
TABLE l I N T E R C O R R E L A T I O N S OF R E S P O N S E S ACROSS S I T U A T I O N S
Conversation probe (CONPROBE) Eye cont Eye cont Sp dur Resp lat ira Sp. dist Smiles Inton Friend
Sp dur
Rsp lat
Sp dist
Smile
Int
Frnd
O-end ?
Inf st
.66**** .62**** .53**** .67**** .73**** .53**** .41"**
Eye cont Sp dur Resp lat ~ Sp dist a. Smiles Z © lnton r..) Friend
O-end Inf st
Key: * p < .05, ** p < .025, *** p < .01, **** p < .001.
of the subjects (randomly selected). Ratings of those behaviors scored on interval scales (e.g., eye contact, response latency) were evaluated by Pearson product-moment correlations. Ratings made on dichotomous occurrence-nonoccurrence scales were evaluated by the following formula: agreements (on occurrences only) divided by agreements plus disagreements, multiplied by 100. Reliabilities for specific response components in all tasks were generally satisfactory; correlations ranged from .78 to 1.00 (X = .93) and percentage agreements ranged from .76 to .90 O? = .84). Reliabilities for overall friendliness ratings were somewhat lower, with correlations ranging from .66 to .79 (X = .71).
Cross-Situational Correspondence Cross-situational correspondence between each role play measure (CIBT and CONPROBE) and each of the other assessment tasks was evaluated with a series of Pearson product-moment correlations. These were calculated between parallel behaviors across situations. The two role play tests shared seven parallel behaviors. The CONPROBE had seven items identical to items in the DYINT, five items identical to items in STINT, and two items identical to items in PSG. A third PSG behavior, Range of Interactions, was considered similar to speech duration on the role play tests in providing some measure of amount of verbal interchange
209
VALIDITY OF ROLE PLAY TESTS TABLE1 CONTINUED Standardized interview
Dyadic interaction (DYINT) Eye cont
Sp dur
Sp dist
Int
Frnd
O-end ?
Inf st
Sp dur
Rsp lat
Sp dist
Problem solving group (PSG)
(STINT) lnt Frnd Smile
Intrpt
Rng inter
Frnd
.46**** .06
.28*
.15 .14
.19
.5l**** .29* .30**
.27* .03
.27*
- .20
.21"*
.30* .29*
.04 .02 .25*
.29* .13
.09 .45***
.24
.15
.32**
.35** .14
in the task. Five CIBT behavioral categories were identical to items in DYINT. The CIBT also had five parallel behaviors in STINT and three parallel behaviors in PSG. The correlational matrix is presented in Table 1. Highly significant relationships were found between CONPROBE and CIBT components. Correlation coefficients ranged from .41 (p < .01) for overall friendliness to .72 (p < .001) for smiles. However, degree of correspondence between behaviors on role plays and criterion situations was by no means extensive. The pattern of relationships between CIBT components and other measures is as follows: eye contact and intonation on CIBT correlated significantly with parallel responses in DYINT; CIBT ratings of speech duration, speech disturbances, intonation and friendliness correlated significantly with parallel responses in STINT; CIBT and PSG ratings of smiles were significantly related. Results of analyses conducted for CONPROBE indicated that ratings of eye contact, speech duration, speech disturbances, and open-ended questions were significantly correlated with parallel DYINT behaviors. Further, CONPROBE scores for speech disturbances and intonation were significantly related to parallel STINT responses. CONPROBE ratings of speech duration were significantly correlated with PSG scores for range of interactions. Also, there was a significant association between overall friendliness ratings across CONPROBE and PSG. Although not
210
VAN HASSELT. HERSEN, AND BELLACK
shown in the table, no significant relationships were found between parallel responses in non-role play tasks.
Test-Retest Reliability of Role Play Tests Test-retest reliability was estimated using the Pearson product-moment correlation. The role play test ( C O N P R O B E and CIBT) was administered to approximately half (20)the subjects 1 week after initial testing. Coefficients for components in each role play variation are presented in Table 2. Correlation coefficients for the CIBT were quite variable ranging from .06 (ns) for intonation to .91 (p < .001) for response latency. By contrast, range of values for C O N P R O B E was more restricted, with coefficients from .54 (p < .009) for smiles to .94 (p < .001) for response latency. In C O N P R O B E , correlation coefficients for two other components (eye contact and overall friendliness) were .80 or greater. Only two CIBT behaviors, smiles and response latency met this criterion.
Correlations between Overall Friendliness Ratings, Teachers' Ratings, Sociometric Scores, and Physical Attractiveness Ratings Pearson product-moment correlation coefficients were computed between ratings of overall friendliness (OF) on behavioral assessment tasks (CIBT, C O N P R O B E , DYINT, S T I N T , and PSG), sociometrics (positive and negative peer nominations, roster and rating scales), physical attractiveness (PA) ratings, and teachers' rating (IBT) items. IBT items of interest included a cumulative score of this measure as well as the following specific items: argues with other children (Item 1), participates well in group (Item 2), makes friends easily (Item 3), defends himself (Item 4), and overall interpersonal skill (Item 5). Significant correlations were found between OF ratings on: CIBT and IBT Item 2 (r = - . 3 5 , p < .025); D Y I N T and Item 1 (r = .49, p < .001), as well as D Y I N T and PA (r = .41, p < .01); S T I N T and Item 3 (r = .31, p < .025); PSG and Item 2 (r = .31, p < .025). H o w e v e r , all other relationships failed to achieve statistical significance; correlations ranged from .01 (OF on C O N P R O B E and negative peer nominations) to .28 (OF on D Y I N T and negative peer nominations) (X = .10). Several significant relationships were found among sociometric measures as well as between these measures and teachers' ratings (see Table 3 below). Positive peer nominations were significantly and negatively correlated with negative peer nominations, roster and rating scores, IBT Item 2, Item 3, and Item 5. Significant correlation coefficients also were found between negative peer nomination ratings and roster and rating scores, IBT cumulative scores, IBT Item 3, Item 5, and PA. Roster and rating scores were significantly related to IBT cumulative scores, IBT Item 1, Item 2, Item 3, and Item 5.
VALIDITY OF ROLE PLAY TESTS
211
TABLE 2 TEST-RETEST RELIABILITY OF ROLE PLAY TESTS CIBT Behavior
Correlations
Significance level
.69 .38 .91 .49 .80 .06 .55
.001 .052 .001 .017 .001 ns .009
Correlations
Significance level
.84 .78 .94 .56 .54 .68 .83 .70 .66
.001 .001 .001 .007 .009 .001 .001 .001 .002
Eye contact Speech duration Response latency Speech disturbance Smiles Intonation Friendliness CONPROBE Behavior Eye contact Speech duration Response latency Speech disturbance Smiles Intonation Friendliness Open ended questions Informative statements
A number of significant interrelationships were found among IBT items and are shown in Table 3.
DISCUSSION Results of this study agree with previous research on validity of role play procedures (e.g., Bellack, Hersen, & Turner, 1978, 1979). In studies with adult populations, a general lack of correspondence between behavior on role play tests and naturalistic situations was found. Similarly, data from the present investigation indicate a number of statistically significant albeit low magnitude correlations between responses in role play tests and criterion situations. While these results suggest that the role play procedure does not have high external validity, the data do not invalidate this test. Several parallel response components were significantly correlated across role play tests and in vivo settings, although the magnitude of the correlations was modest. Thus, there appears to be some relationship between role play and in vivo performance. As Bellack, Hersen, and
~|2
VAN HASSELT,
HERSEN,
AND BELLACK
© t
z
.~
"1 . . . .
<
.< m Z .<
I
,o
e-
I
.<
¢,q ¢.~ .,:~Z 0 0 r~ z
V
I
V Z
o ¢-q V 0~,-"
¢) t4.~
V
~
*
"mo!oos ,s.xoqoeo L
VALIDITY OF ROLE PLAY TESTS
213
Turner (1979) point out, " t h e critical issue is not validity versus invalidity so much as the degree to which the procedure is predictive of behavior in other situations" (p. 676). Somewhat greater transsituational consistency was apparent for the set of items which required subjects to initiate and maintain interactions for a 1-min period (CONPROBE). The typically brief role play format of instruments such as the BAT-R (Eisler et al., 1975) and BAT-C (Bornstein et al., 1977) may be overly restrictive and consequently fail to evoke critical interactive responses. Conversely, " p r o c e d u r e s which require extended interactions might afford a better opportunity for the subject's characteristic response style to e m e r g e " (Bellack, Hersen, & Turner, 1979, p. 676). Also, before definitive statements regarding the validity of role play tests can be made, several other psychometric properties of these instruments must be examined. For example, test reliability, and hence, validity are affected by such factors as test length, item variance, item intercorrelations, sources of error, etc. These traditional parametric influences were not investigated in the present study. Therefore, further research will be needed to address these issues. The present investigation appears to be one of the first to present data regarding the reliability of role play tests. In general, test-retest reliability of the currently employed role play procedures was unacceptable. Considerable variability was evidenced on CIBT coefficients (range = .06 to .91). Further, only two of these values (response latency and smiles) were .80 or greater, although four of seven coefficients achieved statistical significance. Less variability was shown for C O N P R O B E components (range = .54 to .94). H o w e v e r , on this device, only three of nine coefficients (eye contact, response latency, and overall friendliness) were greater than .80. Possible reasons for these results include (1) relatively low initial inter-rater agreements for some behaviors (e.g., overall fi'iendliness), which make it unlikely that test-retest coefficients will be acceptable and (2) the likelihood that the anxiety-producing qualities of role play tests serve as a potential constraint on the reliability (cf. Hersen, Bellack, & Turner, 1978). Regarding the latter point, anxiety effects of the analogue measure may have been attenuated upon second testing due to habituation or familiarity. H o w e v e r , further analyses indicated role play performance on second administration was no more correlated with criterion measures than was performance on first administration. The relationship between children's social (sociometric) status and their interpersonal behavior, as assessed on several social performance measures, was examined. Results indicated virtually no correspondence between scores on sociometric and global ratings of friendliness in role play and naturalistic situations. A number of significant relationships were found between sociometric scores and teachers' ratings of children's social skill. H o w e v e r , correspondence between these measures and overall friendliness ratings on role play and criterion situations was generally low. Reardon et al. (1979)
214
VAN HASSELT, HERSEN, AND BELLACK
also found low correlations between teachers' ratings of children's assertiveness and this same dimension assessed on role play tests. One explanation for the results in the present study concerns the possibility that there are other, more relevant factors (than degree of "appearing friendly") to which children and teachers may be responding. Asher, Oden, and Gottman (1977) review a number of determinants of peer acceptance (e.g., personal and situational characteristics, accurate communication, being expert). For example, in an earlier study by McGraw and Tolbert (1953), a highly significant correlation was found between athletic ability (i.e., being expert) and social status in junior high school boys. This and numerous other studies (see review by Asher et al., 1977) suggest that social skills researchers may be attending to comparatively irrelevant variables. Perhaps teaching children to be expert softball or basketball players would enhance their social status more than training on eye contact, intonation, open-ended questions, etc. Clearly, more empirical data are needed to determine variables of consequence in children's social functioning.
REFERENCE NOTES 1. Beck, S., & Forehand, R. Social skills training for children: A review and methodological analysis o f behavior modification studies. Unpublished manuscript, University of Georgia, 1978. 2. Hymel, S., & Asher, S.R. Assessment and training o f isolated children's social skills. Paper presented at the biennial meeting of the Society for Research in Child Development, New Orleans, March 1977. 3. Singleton, L. C., & Asher, S. R. Sociometric ratings and social interaction among third grade children in an integrated school district. Unpublished manuscript, University of Illinois, 1976. 4. Marsh, D. T., Serafica, F. C., & Barenboim, C. Interrelationships among perspectivetaking, interpersonal problem solving, and interpersonal functioning. Unpublished manuscript, University of Pittsburgh, 1977.
REFERENCES Asher, S. R., Oden, S. L., & Gottman, J . M . Children's friendships in school settings. In L. G. Katz (Ed.), Current topics in early childhood education (Vol. 1). Norwood, N J: Ablex Publishing, 1977. Bellack, A. S. Behavioral assessment of social skills. In A. S. Bellack & M. Hersen (Eds.), Research and practice in social skills training. New York: Plenum, 1979. Bellack, A. S., Hersen, M., & Lamparski, D. Role play tests for assessing social skills: Are they valid? Are they useful? Journal o f Consulting and ClinicalPsychology, 1979, 47, 335-342. Bellack, A. S., Hersen, M., & Turner, S . M . Role play tests for assessing social skills: Are they valid? Behavior Therapy, 1978, 9, 448-461. Bellack, A. S., Hersen, M., & Turner, S . M . The relationship of role playing and k n o w l -
VALIDITY OF ROLE PLAY TESTS
215
edge of appropriate behavior to assertion in the natural environment. Journal of Consulting and Clinical Psychology, 1979, 45, 679-685. Bornstein, M. R., Bellack, A. S., & Hersen, M. Social skills training for unassertive children: A multiple baseline analysis. Journal of Applied Behavior Analysis, 1977, 10, 183-195. Bornstein, M. R., Bellack, A. S., & Hersen, M. Socially aggressive training for highly aggressive children in an inpatient psychiatric setting. Behavior Modification, 1980, 4, 173-186. Combs, M. L., & Slaby, D . A . Social skills training with children. In B. B. Lahey & A. E. Kazdin (Eds.), Advances in clinical child psychology: Volume I. New York: Plenum, 1977. Cowen, E. L., Pederson, A., Babigian, H., Izzo, L. D., & Trost, M . A . Long-term followup of early detected vulnerable children. Journal of Consulting and Clinical Psychology, 1973, 41,438-446. Criswell, J. H. A sociometric study of race in the classroom. Archives of Psychology, 1939, 235, 1-82. Curran, J . P . Comments on BeUack, Hersen, and Turner's paper on the validity of role play test. Behavior Therapy, 1978, 9, 462-468. Eisler, R. M., Hersen, M., & Miller, P.M. Effects of modeling on components of assertive behavior. Journal of Behavior Therapy and Experimental Psychiatry, 1973, 4, 1-6. Eisler, R. M., Hersen, M., Miller, P. M., & Blanchard, E . B . Situational determinants of assertive behavior. Journal of Consulting and Clinical Psychology, 1975, 43, 330-340. Goldfried, M. R., & Kent, R. N. Traditional vs. behavioral personality assessment: A comparison of methodological & theoretical assumptions. PsychologicalBulletin, 1972, 77, 409-420. Hersen, M., Bellack, A. S., & Turner, S. M. Assessment of assertiveness in female psychiatric patients: Motor and autonomic measures. Journal of Behavior Therapy and Experimental Psychiatry, 1978, 9, 11-16. Kagan, J., & Moss, H . A . Birth to maturity: A study in psychological development. New York: Wiley, 1962. Keller, H. R., D e y s a c h , R. E., Hiers, T. G., Ross, A. S., & Moore, L. L. Interrelationships among decentering and behavioral ratings in a therapeutic camp setting. Journal of Genetic Psychology, 1976, 129, 113-120. McGraw, L. W., & Tolbert, J . W . Sociometric status and athletic ability of junior high school boys. The Research Quarterly, 1953, 24, 72-80. Michelson, L., & Wood, R. Behavioral assessment and training of children's social skills. In M. Hersen, R. M. Eisler, & P. M. Miller (Eds.), Progress in behavior modification: Volume 8. New York: Academic Press, in press. Mischel, W. Direct versus indirect personality assessment: Evidence and implications. Journal of Consulting a~d Clinical Psychology, 1972, 38, 319-324. Reardon, R. C., Hersen, M., Bellack, A. S., & Foley, J. M. Measuring social skill in grade school boys. Journal of Behavioral Assessment, 1979, 1, 87-105. Roff, M. Childhood social interactions and young adult bad conduct. Journal of Abnormal and Social Psychology, 1961, 63, 333-337. Rosenfeld, H. M. Effect of an approval-seeking induction on interpersonal proximity. Psychological Reports, 1965, 17, 120-122. Shaftel, F. R., & Shaftel, G. Role playing for social values: Decision-making in the social studies. Englewood Cliffs, NJ: Prentice-Hall, 1967. Van Hasselt, V. B., Hersen, M., Whitehill, M. B., & Bellack, A.S. Social skill assessment and training for children: An evaluative review. Behaviour Research and Therapy, 1979, 17, 413-437.
216
VAN HASSELT, HERSEN, AND BELLACK
Whitehill, M. B., Hersen, M., & Bellack, A. S. A conversation skills training program for socially isolated children: An analysis of generalization. Behaviour Research and Therapy, 1980, 18, 217-225. RECEIVED: May 16, 1980; REVISED" July 22, 1980 FINAL ACCEPTANCE" September 15, 1980
The Validity of Role Play Tests for Assessing Social Skills in Children VINCENT B . VAN HASSELT
University of Pittsburgh MICHEL HERSEN
Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine ALAN S. BELLACK
University of Pittsburgh
A correlational study was conducted in an effort to examine the validity and reliability of role play tests for assessing social skills in children. Twenty-two male and twenty female elementary school children, ranging in age from 8-10 years, responded to role play scenarios involving social interactions. In addition to the role play procedure, a number of other assessment strategies were utilized: (1) observations of behavior in naturalistic settings, (2) sociometric ratings, and (3) teachers' ratings. Results did not support the validity of the role play test. Correlations between role play performance and criterion measures were generally quite low. Test-retest reliability of the role play tests also was unacceptable. Further, results indicated that while several significant relationships were found between children's sociometric scores and teachers' ratings of interpersonal behavior, little association was found between these measures and global ratings of friendliness in role play and in vivo situations. Results are discussed in terms of
This article is based on the senior author's Masters Thesis, conducted under the direction of the second and third authors, and submitted to the Graduate Faculty of Arts and Sciences, University of Pittsburgh, in partial fulfillment of the requirements for the Master of Science degree. Portions of this paper were presented at the 87th Annual Convention of the American Psychological Association, New York, NY, September, 1979. The authors thank Denise A. Herbol, Curtis Gibson, Steven Stern, Denise Reid, Ronald Barbati, and Carla Costanza for their varied contributions throughout this project and Susan Giusti, Cheryl Kubelick, Bill McDonald, and Tim Rusnak of the Falk Elementary School for their extensive cooperation. Requests for reprints should be sent to Michel Hersen, Department of Psychiatry, Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine, Pittsburgh, PA 15261. 202 0005-7894/81/0202-021651.00/0 Copyright 1981 by Association for Advancement of Behavior Therapy All rights of reproduction in any form reserved.
VALIDITY OF ROLE PLAY TESTS
203
the validity and utility of role play tests for children, the relationship of these data to previous research ~n social skills assessment, and the heuristic value of examining research concerning children's interpersonal functioning from developmental and social psychological perspectives.
Increased attention has been directed to the assessment and remediation of interpersonal skill deficiencies in children (see reviews by Beck & Forehand, Note 1; Combs & Slaby, 1977; Michelson & Wood, in press; Van Hasselt, Hersen, Whitehill, & Bellack, 1979). This growing interest is attributed to a large body of research demonstrating an association between level of children's social functioning and their long-term adjustment (cf. Cowen, Pederson, Babigian, Izzo, & Trost, 1973; Kagan & Moss, 1962; Roff, 1961). These findings underscore the need to develop effective interventions for children who demonstrate poor social competence (i.e., aggression, unassertiveness, social withdrawal). However, to evaluate adequately the efficacy of treatment regimens applied to this population, employment of valid and reliable assessment strategies is essential. While behavior therapists generally have favored in vivo assessment methods (Goldfried & Kent, 1972; Mischel, 1972), issues of ethics, economics, and pragmatics often preclude use of such strategies. Thus, analogue strategies (e.g., behavioral avoidance tasks, role play tests) have frequently been used as viable alternatives. For example, in the area of social skills research, role play tests "have become the hallmark o f . . . assessment" (Bellack, 1979). Regardless of how appealing use of such procedures may be (they are convenient, economical, face valid), their value is questionable if they are psychometrically unsound. Recently, Bellack and his colleagues (Bellack, Hersen, & Lamparski, 1979; Bellack, Hersen, & Turner, 1978, 1979) found that role play procedures were n o t externally valid for chronic psychiatric patients and college students (i.e., little correspondence was found between role play performances and interpersonal behavior in more naturalistic settings). Reardon, Hersen, Bellack, and Foley (1979) presented the first data concerning the relationship between children's role played assertive behavior and their responses in other social performance tasks. Results indicated that children classified as high assertive or low assertive on the basis of overall ratings differed significantly on several behavioral response components (e.g., response latency, speech duration). However, behavioral ratings of overall assertiveness showed little relationship to self-report or teacher ratings of assertiveness. While the study by Reardon et al. (1979) provides important preliminary data pertaining to children's interpersonal behavior, the external validity of their primary assessment tool (i.e., role play test) has yet to be determined. The primary purpose of our study is to determine the external validity of role play tests for assessing social skills in children. Correspondence between response components in children's role play
204
VAN H A S S E L T , H E R S E N , A N D B E L L A C K
tests and in more naturalistic situations is examined. A second purpose of this investigation is to examine the relationship between behavioral assessment and sociometric status of children.
METHOD Subjects Twenty-two males and twenty females between the ages of 8 and 10 were selected at random from class rosters of the Falk School of the University of Pittsburgh. The male group included six third-graders, eight fourth-graders, and eight fifth-graders. The female group included four third-graders, eight fourth-graders, and eight fifth-graders. The children were informed of the voluntary status of their participation. Consent was obtained via form letters sent to parents which described procedures and objectives of the project. Procedure The procedure involved observation of children's interpersonal behavior in four different settings: (1) problem solving groups, (2) standardized interviews, (3) role play tests, and (4) dyadic interactions. Sociometric instruments and teachers' rating scales also were administered. All subjects were first involved in problem solving groups which were conducted at the Falk School. Standardized interviews (also administered at Falk) as well as role play tests and dyadic interactions (both conducted at the Clinical Psychology Center of the University of Pittsburgh) were carried out approximately 1 week after problem solving sessions. Dyadic interactions immediately preceded administration of role play tests. Onehalf the subjects participated in the dyadic interactions and role play tests first, while the other half received the standardized interview first. Administration of the remainder of the tasks was scheduled according to the availability of subjects and/or teachers, with the restriction that no two tasks could be administered on the same day.Role play tests. Two forms of role play tests for children were employed in the present investigation: (1) the Children's Interpersonal Behavior Test (CIBT) and (2) the Conversation Probe (CONPROBE) (Whitehill, Hersen, & Bellack, 1980). 1 (1) CIBT. The CIBT is a modified version of the Behavioral Assertiveness Test for Boys (BAT-B) (Reardon et al., 1979) and the Behavioral Assertiveness Test for Children (BAT-C) (Bornstein, Bellack, & Hersen, 1977). It consists of six role play situations designed to be typical of social situations children are likely to encounter in their natural environments. Teacher input and feedback regarding realism and representativeness of scene content were obtained during item construction. Half the scenes in 1Copies of the CIBT and CONPROBE are available upon request from Vincent B. Van Hasselt, Clinical PsychologyCenter, University of Pittsburgh, Pittsburgh, PA 15260.
VALIDITY
OF ROLE PLAY TESTS
205
the CIBT involved a male role play partner while the other half involved a female role play partner. Unlike the Behavioral Assertiveness Test--Revised (BAT-R) (Eisler, Hersen, Miller, & Blanchard, 1975) or current behavioral assertiveness tests for children (in which termination of the subject's response to the prompt serves as a cue for the experimenter to present the next situation), the role play partner in the CIBT followed the subject's initial response with two additional prearranged counterresponses to allow for an extended interchange. This format was designed to make CIBT items more similar to real-life encounters, which often are longer than interactions possible with one-line prompts and responses on instruments such as the BAT-R (Curran, 1978). (2) CONPROBE. To assess extended interactions necessary for initiating a conversation Whitehill et al. (1980) developed the CONPROBE. This assessment device consists of single interpersonal situations similar in format to those found in the CIBT. However, the CONPROBE enables the subject to engage in prolonged (1-rnin) conversations with the role model. Also, with the CONPROBE, the confederate does not deliver a prompt. Rather, the child is asked to initiate and maintain an interaction following description of the interpersonal situation. Confederates were directed to respond to subjects in a facilitating, nondominating manner, and were asked to break silences of longer than 5 sec by introduction of a brief, facilitative question. Procedures for administration of CIBT and CONPROBE were similar to those described in previous social skills investigations with children (see Bornstein, Bellack, & Hersen, 1977, 1980; Reardon et al., 1979). Role play partners (one male and one female) for both the CIBT and CONPROBE were undergraduate psychology majors who received approximately 8 hours of training in these procedures. The same set of role play partners was used for both tasks. A male graduate student in clinical psychology served as narrator for administration of the role play test. Two additional undergraduates were involved in the rating of behavioral components, while another two were involved in independent ratings of overall friendliness on role plays. Scoring. Videotaped responses to CONPROBE and CIBT were subsequently rated on behavioral components of social skill identified by Eisler and his colleagues (Eisler, Hersen, & Miller, 1973; Eisler et al., 1975) and Whitehill et al. (1980). Overall friendliness also was rated on these tasks. Several other assessment instruments were employed in this investigation and are described in the following sections. Dyadic interactions (DYINT). Dyads consisting of two experimental subjects were observed in a waiting room situation. Twenty dyads included one male and one female subject. One dyad consisted of two males. When subjects scheduled for a particular session arrived at the Clinical Psychology Center, they were led to a room in groups of two and informed that their task (the role play test) would be administered in a
206
VAN H A S S E L T , H E R S E N , A N D B E L L A C K
few minutes. They were further asked to stay seated and told that they were free to converse until it was time to begin. The observation room was empty except for a pair of chairs about 5 ft apart, facing each other at an angle of about 90°. Such positioning was suggested by Rosenfeld (1965) to constitute a "neutral affiliative distance." It has the advantage of facilitating observation of nonverbal behaviors. Dyadic interactions were of 5-min duration. They were videotaped through a one-way mirror and rated for eye contact, speech duration, intonation, voice volume, speech disturbances, smiles, and overall friendliness. Standardized interview (SI). Each subject was individually interviewed by a female undergraduate research assistant who was not involved in administration of the other procedures. She was trained to administer this task in a uniform manner. Subjects in the present study were asked six open-ended questions (e.g., How are you today? How do you like school this year?). Interview sessions were audiotaped and subsequently rated for response latency (to questions), speech duration, intonation, voice volume and overall friendliness. Problem solving group (PSG). Subjects were observed participating in an open-ended interpersonal interaction which required them to analyze and produce a solution for an interpersonal dilemma. The problem, adapted from Shaftel and Shaftel (1967), presented children with a situation in which they had to decide which of three friends is to pay for a lamp broken when the youngest of the three hits it with his frisbee. This occurred after the child's mother told the older children to play frisbee outdoors. Groups consisted of three children randomly selected from the pool of experimental subjects. The experimenter first described the dilemma to the group. Following description of the problem, subjects were asked (as a group) a series of questions: (a) What is the problem here? (b) What are all the possible ways to solve this problem? (c) What might happen if each of these solutions is used? (d) Which solution is best? A drawing consisting of all the characters in the problem was provided to the group for reference purposes. Group interactions were 16 min in duration and were observed by trained raters. An interval recording procedure was utilized in which each child was alternately observed in 2-min blocks (eight 15-sec intervals) by two raters for occurrence or nonoccurrence of the following response components: smiles, interruptions, and range of interactions. Observers were undergraduates whose training continued until a criterion of 80% inter-rater agreement was reached. Each of the two raters was assigned to observe two children in a session. In all cases, there was overlap of one child in behavioral observations which served as a reliability check. Thus, reliability ratings were based on observations of approximately 33% of the sample. Sociometrics. The sociometric assessment method used in this study was adapted from Hymel and Asher (Note 2). (1) Positive peer nomination sociometric. Each child was given a list of all his/her classmates and asked to circle the names of three children
VALIDITY OF ROLE PLAY TESTS
207
they "especially like in s c h o o l . " Children were provided with a class roster to avoid the possibility that a child was not nominated because he/ she was forgotten. (2) Negative peer nomination sociometric. Each child was given a list of his/her classmates and asked to circle the names of three children he/she "especially did not like in school." Class rosters were again provided to students to minimize omissions. Subjects' scores on the above measures consisted of the number o f nominations received from same-sex peers. Only same-sex nominations were used since previous research has shown that children of this age group typically give low ratings to opposite-sex peers (Criswell, 1939; Singleton & Asher, Note 3). (3) Roster and rating scale sociometric. Children were asked to give ratings of each classmate on a 1 to 5 scale in answer to the question, " H o w much do you like to play with this person at s c h o o l ? " A rating of 1 indicated that the child did not like to play with that classmate. Children were provided with an alphabetized list of their classmates on which each name was followed by the numbers 1, 2, 3, 4, and 5. A child's score on the play ratings consisted of the average rating received from same-sex peers.
Teacher's checklist of interpersonal behavior (IBT) (Keller, Deysach, Hiers, Ross, & Moore, 1976). The IBT is a measure of interpersonal functioning in children. This instrument includes 17 items which can be adequately assessed within the school context. Some of these items include: "fights with other children," "complies with teacher's r e q u e s t s , " and "participates well in group activities." An additional item, used by Marsh, Serafica, and Barenboim (Note 4) included in the present study was "overall level of interpersonal skill relative to other children of the same a g e . " Classroom teachers rated each subject on each of the first 16 items using the following six categories: almost always, very often, often, sometimes, rarely, and never. In addition, subjects were rated with reference to the final item (overall interpersonal effectiveness) as follows: top 5%, top 25%, middle 25%, lowest 25%, and lowest 5%. The first 16 items were scored 1 to 6 (never to almost always) to represent the six possible categories (lowest 5% to top 5%). Thus, there were 17 individual scores for each subject. Physical attractiveness ratings (PA). Four additional undergraduates (three females, one male) independently rated each subject on physical attractiveness. Each judge observed a 30-sec video segment (no audio) from the middle of an individual subject's role play and employed a 5point scale (from 1 = very unattractive to 5 = very attractive).
RESULTS Reliability of Ratings Reliability (agreement) for ratings of each measure was assessed by comparing the scores of a primary rater with a reliability rater for 25%
208
VAN HASSELT,
HERSEN,
AND BELLACK
TABLE l I N T E R C O R R E L A T I O N S OF R E S P O N S E S ACROSS S I T U A T I O N S
Conversation probe (CONPROBE) Eye cont Eye cont Sp dur Resp lat ira Sp. dist Smiles Inton Friend
Sp dur
Rsp lat
Sp dist
Smile
Int
Frnd
O-end ?
Inf st
.66**** .62**** .53**** .67**** .73**** .53**** .41"**
Eye cont Sp dur Resp lat ~ Sp dist a. Smiles Z © lnton r..) Friend
O-end Inf st
Key: * p < .05, ** p < .025, *** p < .01, **** p < .001.
of the subjects (randomly selected). Ratings of those behaviors scored on interval scales (e.g., eye contact, response latency) were evaluated by Pearson product-moment correlations. Ratings made on dichotomous occurrence-nonoccurrence scales were evaluated by the following formula: agreements (on occurrences only) divided by agreements plus disagreements, multiplied by 100. Reliabilities for specific response components in all tasks were generally satisfactory; correlations ranged from .78 to 1.00 (X = .93) and percentage agreements ranged from .76 to .90 O? = .84). Reliabilities for overall friendliness ratings were somewhat lower, with correlations ranging from .66 to .79 (X = .71).
Cross-Situational Correspondence Cross-situational correspondence between each role play measure (CIBT and CONPROBE) and each of the other assessment tasks was evaluated with a series of Pearson product-moment correlations. These were calculated between parallel behaviors across situations. The two role play tests shared seven parallel behaviors. The CONPROBE had seven items identical to items in the DYINT, five items identical to items in STINT, and two items identical to items in PSG. A third PSG behavior, Range of Interactions, was considered similar to speech duration on the role play tests in providing some measure of amount of verbal interchange
209
VALIDITY OF ROLE PLAY TESTS TABLE1 CONTINUED Standardized interview
Dyadic interaction (DYINT) Eye cont
Sp dur
Sp dist
Int
Frnd
O-end ?
Inf st
Sp dur
Rsp lat
Sp dist
Problem solving group (PSG)
(STINT) lnt Frnd Smile
Intrpt
Rng inter
Frnd
.46**** .06
.28*
.15 .14
.19
.5l**** .29* .30**
.27* .03
.27*
- .20
.21"*
.30* .29*
.04 .02 .25*
.29* .13
.09 .45***
.24
.15
.32**
.35** .14
in the task. Five CIBT behavioral categories were identical to items in DYINT. The CIBT also had five parallel behaviors in STINT and three parallel behaviors in PSG. The correlational matrix is presented in Table 1. Highly significant relationships were found between CONPROBE and CIBT components. Correlation coefficients ranged from .41 (p < .01) for overall friendliness to .72 (p < .001) for smiles. However, degree of correspondence between behaviors on role plays and criterion situations was by no means extensive. The pattern of relationships between CIBT components and other measures is as follows: eye contact and intonation on CIBT correlated significantly with parallel responses in DYINT; CIBT ratings of speech duration, speech disturbances, intonation and friendliness correlated significantly with parallel responses in STINT; CIBT and PSG ratings of smiles were significantly related. Results of analyses conducted for CONPROBE indicated that ratings of eye contact, speech duration, speech disturbances, and open-ended questions were significantly correlated with parallel DYINT behaviors. Further, CONPROBE scores for speech disturbances and intonation were significantly related to parallel STINT responses. CONPROBE ratings of speech duration were significantly correlated with PSG scores for range of interactions. Also, there was a significant association between overall friendliness ratings across CONPROBE and PSG. Although not
210
VAN HASSELT. HERSEN, AND BELLACK
shown in the table, no significant relationships were found between parallel responses in non-role play tasks.
Test-Retest Reliability of Role Play Tests Test-retest reliability was estimated using the Pearson product-moment correlation. The role play test ( C O N P R O B E and CIBT) was administered to approximately half (20)the subjects 1 week after initial testing. Coefficients for components in each role play variation are presented in Table 2. Correlation coefficients for the CIBT were quite variable ranging from .06 (ns) for intonation to .91 (p < .001) for response latency. By contrast, range of values for C O N P R O B E was more restricted, with coefficients from .54 (p < .009) for smiles to .94 (p < .001) for response latency. In C O N P R O B E , correlation coefficients for two other components (eye contact and overall friendliness) were .80 or greater. Only two CIBT behaviors, smiles and response latency met this criterion.
Correlations between Overall Friendliness Ratings, Teachers' Ratings, Sociometric Scores, and Physical Attractiveness Ratings Pearson product-moment correlation coefficients were computed between ratings of overall friendliness (OF) on behavioral assessment tasks (CIBT, C O N P R O B E , DYINT, S T I N T , and PSG), sociometrics (positive and negative peer nominations, roster and rating scales), physical attractiveness (PA) ratings, and teachers' rating (IBT) items. IBT items of interest included a cumulative score of this measure as well as the following specific items: argues with other children (Item 1), participates well in group (Item 2), makes friends easily (Item 3), defends himself (Item 4), and overall interpersonal skill (Item 5). Significant correlations were found between OF ratings on: CIBT and IBT Item 2 (r = - . 3 5 , p < .025); D Y I N T and Item 1 (r = .49, p < .001), as well as D Y I N T and PA (r = .41, p < .01); S T I N T and Item 3 (r = .31, p < .025); PSG and Item 2 (r = .31, p < .025). H o w e v e r , all other relationships failed to achieve statistical significance; correlations ranged from .01 (OF on C O N P R O B E and negative peer nominations) to .28 (OF on D Y I N T and negative peer nominations) (X = .10). Several significant relationships were found among sociometric measures as well as between these measures and teachers' ratings (see Table 3 below). Positive peer nominations were significantly and negatively correlated with negative peer nominations, roster and rating scores, IBT Item 2, Item 3, and Item 5. Significant correlation coefficients also were found between negative peer nomination ratings and roster and rating scores, IBT cumulative scores, IBT Item 3, Item 5, and PA. Roster and rating scores were significantly related to IBT cumulative scores, IBT Item 1, Item 2, Item 3, and Item 5.
VALIDITY OF ROLE PLAY TESTS
211
TABLE 2 TEST-RETEST RELIABILITY OF ROLE PLAY TESTS CIBT Behavior
Correlations
Significance level
.69 .38 .91 .49 .80 .06 .55
.001 .052 .001 .017 .001 ns .009
Correlations
Significance level
.84 .78 .94 .56 .54 .68 .83 .70 .66
.001 .001 .001 .007 .009 .001 .001 .001 .002
Eye contact Speech duration Response latency Speech disturbance Smiles Intonation Friendliness CONPROBE Behavior Eye contact Speech duration Response latency Speech disturbance Smiles Intonation Friendliness Open ended questions Informative statements
A number of significant interrelationships were found among IBT items and are shown in Table 3.
DISCUSSION Results of this study agree with previous research on validity of role play procedures (e.g., Bellack, Hersen, & Turner, 1978, 1979). In studies with adult populations, a general lack of correspondence between behavior on role play tests and naturalistic situations was found. Similarly, data from the present investigation indicate a number of statistically significant albeit low magnitude correlations between responses in role play tests and criterion situations. While these results suggest that the role play procedure does not have high external validity, the data do not invalidate this test. Several parallel response components were significantly correlated across role play tests and in vivo settings, although the magnitude of the correlations was modest. Thus, there appears to be some relationship between role play and in vivo performance. As Bellack, Hersen, and
~|2
VAN HASSELT,
HERSEN,
AND BELLACK
© t
z
.~
"1 . . . .
<
.< m Z .<
I
,o
e-
I
.<
¢,q ¢.~ .,:~Z 0 0 r~ z
V
I
V Z
o ¢-q V 0~,-"
¢) t4.~
V
~
*
"mo!oos ,s.xoqoeo L
VALIDITY OF ROLE PLAY TESTS
213
Turner (1979) point out, " t h e critical issue is not validity versus invalidity so much as the degree to which the procedure is predictive of behavior in other situations" (p. 676). Somewhat greater transsituational consistency was apparent for the set of items which required subjects to initiate and maintain interactions for a 1-min period (CONPROBE). The typically brief role play format of instruments such as the BAT-R (Eisler et al., 1975) and BAT-C (Bornstein et al., 1977) may be overly restrictive and consequently fail to evoke critical interactive responses. Conversely, " p r o c e d u r e s which require extended interactions might afford a better opportunity for the subject's characteristic response style to e m e r g e " (Bellack, Hersen, & Turner, 1979, p. 676). Also, before definitive statements regarding the validity of role play tests can be made, several other psychometric properties of these instruments must be examined. For example, test reliability, and hence, validity are affected by such factors as test length, item variance, item intercorrelations, sources of error, etc. These traditional parametric influences were not investigated in the present study. Therefore, further research will be needed to address these issues. The present investigation appears to be one of the first to present data regarding the reliability of role play tests. In general, test-retest reliability of the currently employed role play procedures was unacceptable. Considerable variability was evidenced on CIBT coefficients (range = .06 to .91). Further, only two of these values (response latency and smiles) were .80 or greater, although four of seven coefficients achieved statistical significance. Less variability was shown for C O N P R O B E components (range = .54 to .94). H o w e v e r , on this device, only three of nine coefficients (eye contact, response latency, and overall friendliness) were greater than .80. Possible reasons for these results include (1) relatively low initial inter-rater agreements for some behaviors (e.g., overall fi'iendliness), which make it unlikely that test-retest coefficients will be acceptable and (2) the likelihood that the anxiety-producing qualities of role play tests serve as a potential constraint on the reliability (cf. Hersen, Bellack, & Turner, 1978). Regarding the latter point, anxiety effects of the analogue measure may have been attenuated upon second testing due to habituation or familiarity. H o w e v e r , further analyses indicated role play performance on second administration was no more correlated with criterion measures than was performance on first administration. The relationship between children's social (sociometric) status and their interpersonal behavior, as assessed on several social performance measures, was examined. Results indicated virtually no correspondence between scores on sociometric and global ratings of friendliness in role play and naturalistic situations. A number of significant relationships were found between sociometric scores and teachers' ratings of children's social skill. H o w e v e r , correspondence between these measures and overall friendliness ratings on role play and criterion situations was generally low. Reardon et al. (1979)
214
VAN HASSELT, HERSEN, AND BELLACK
also found low correlations between teachers' ratings of children's assertiveness and this same dimension assessed on role play tests. One explanation for the results in the present study concerns the possibility that there are other, more relevant factors (than degree of "appearing friendly") to which children and teachers may be responding. Asher, Oden, and Gottman (1977) review a number of determinants of peer acceptance (e.g., personal and situational characteristics, accurate communication, being expert). For example, in an earlier study by McGraw and Tolbert (1953), a highly significant correlation was found between athletic ability (i.e., being expert) and social status in junior high school boys. This and numerous other studies (see review by Asher et al., 1977) suggest that social skills researchers may be attending to comparatively irrelevant variables. Perhaps teaching children to be expert softball or basketball players would enhance their social status more than training on eye contact, intonation, open-ended questions, etc. Clearly, more empirical data are needed to determine variables of consequence in children's social functioning.
REFERENCE NOTES 1. Beck, S., & Forehand, R. Social skills training for children: A review and methodological analysis o f behavior modification studies. Unpublished manuscript, University of Georgia, 1978. 2. Hymel, S., & Asher, S.R. Assessment and training o f isolated children's social skills. Paper presented at the biennial meeting of the Society for Research in Child Development, New Orleans, March 1977. 3. Singleton, L. C., & Asher, S. R. Sociometric ratings and social interaction among third grade children in an integrated school district. Unpublished manuscript, University of Illinois, 1976. 4. Marsh, D. T., Serafica, F. C., & Barenboim, C. Interrelationships among perspectivetaking, interpersonal problem solving, and interpersonal functioning. Unpublished manuscript, University of Pittsburgh, 1977.
REFERENCES Asher, S. R., Oden, S. L., & Gottman, J . M . Children's friendships in school settings. In L. G. Katz (Ed.), Current topics in early childhood education (Vol. 1). Norwood, N J: Ablex Publishing, 1977. Bellack, A. S. Behavioral assessment of social skills. In A. S. Bellack & M. Hersen (Eds.), Research and practice in social skills training. New York: Plenum, 1979. Bellack, A. S., Hersen, M., & Lamparski, D. Role play tests for assessing social skills: Are they valid? Are they useful? Journal o f Consulting and ClinicalPsychology, 1979, 47, 335-342. Bellack, A. S., Hersen, M., & Turner, S . M . Role play tests for assessing social skills: Are they valid? Behavior Therapy, 1978, 9, 448-461. Bellack, A. S., Hersen, M., & Turner, S . M . The relationship of role playing and k n o w l -
VALIDITY OF ROLE PLAY TESTS
215
edge of appropriate behavior to assertion in the natural environment. Journal of Consulting and Clinical Psychology, 1979, 45, 679-685. Bornstein, M. R., Bellack, A. S., & Hersen, M. Social skills training for unassertive children: A multiple baseline analysis. Journal of Applied Behavior Analysis, 1977, 10, 183-195. Bornstein, M. R., Bellack, A. S., & Hersen, M. Socially aggressive training for highly aggressive children in an inpatient psychiatric setting. Behavior Modification, 1980, 4, 173-186. Combs, M. L., & Slaby, D . A . Social skills training with children. In B. B. Lahey & A. E. Kazdin (Eds.), Advances in clinical child psychology: Volume I. New York: Plenum, 1977. Cowen, E. L., Pederson, A., Babigian, H., Izzo, L. D., & Trost, M . A . Long-term followup of early detected vulnerable children. Journal of Consulting and Clinical Psychology, 1973, 41,438-446. Criswell, J. H. A sociometric study of race in the classroom. Archives of Psychology, 1939, 235, 1-82. Curran, J . P . Comments on BeUack, Hersen, and Turner's paper on the validity of role play test. Behavior Therapy, 1978, 9, 462-468. Eisler, R. M., Hersen, M., & Miller, P.M. Effects of modeling on components of assertive behavior. Journal of Behavior Therapy and Experimental Psychiatry, 1973, 4, 1-6. Eisler, R. M., Hersen, M., Miller, P. M., & Blanchard, E . B . Situational determinants of assertive behavior. Journal of Consulting and Clinical Psychology, 1975, 43, 330-340. Goldfried, M. R., & Kent, R. N. Traditional vs. behavioral personality assessment: A comparison of methodological & theoretical assumptions. PsychologicalBulletin, 1972, 77, 409-420. Hersen, M., Bellack, A. S., & Turner, S. M. Assessment of assertiveness in female psychiatric patients: Motor and autonomic measures. Journal of Behavior Therapy and Experimental Psychiatry, 1978, 9, 11-16. Kagan, J., & Moss, H . A . Birth to maturity: A study in psychological development. New York: Wiley, 1962. Keller, H. R., D e y s a c h , R. E., Hiers, T. G., Ross, A. S., & Moore, L. L. Interrelationships among decentering and behavioral ratings in a therapeutic camp setting. Journal of Genetic Psychology, 1976, 129, 113-120. McGraw, L. W., & Tolbert, J . W . Sociometric status and athletic ability of junior high school boys. The Research Quarterly, 1953, 24, 72-80. Michelson, L., & Wood, R. Behavioral assessment and training of children's social skills. In M. Hersen, R. M. Eisler, & P. M. Miller (Eds.), Progress in behavior modification: Volume 8. New York: Academic Press, in press. Mischel, W. Direct versus indirect personality assessment: Evidence and implications. Journal of Consulting a~d Clinical Psychology, 1972, 38, 319-324. Reardon, R. C., Hersen, M., Bellack, A. S., & Foley, J. M. Measuring social skill in grade school boys. Journal of Behavioral Assessment, 1979, 1, 87-105. Roff, M. Childhood social interactions and young adult bad conduct. Journal of Abnormal and Social Psychology, 1961, 63, 333-337. Rosenfeld, H. M. Effect of an approval-seeking induction on interpersonal proximity. Psychological Reports, 1965, 17, 120-122. Shaftel, F. R., & Shaftel, G. Role playing for social values: Decision-making in the social studies. Englewood Cliffs, NJ: Prentice-Hall, 1967. Van Hasselt, V. B., Hersen, M., Whitehill, M. B., & Bellack, A.S. Social skill assessment and training for children: An evaluative review. Behaviour Research and Therapy, 1979, 17, 413-437.
216
VAN HASSELT, HERSEN, AND BELLACK
Whitehill, M. B., Hersen, M., & Bellack, A. S. A conversation skills training program for socially isolated children: An analysis of generalization. Behaviour Research and Therapy, 1980, 18, 217-225. RECEIVED: May 16, 1980; REVISED" July 22, 1980 FINAL ACCEPTANCE" September 15, 1980