- Email: [email protected]
The influence of test scores and naturally-occurring pupil characteristics on psychoeducational decision making with children
Journal of School Psychology 1981"Vol. 19, No. 2
0022-4405/81/1400-0167500.95 ©The Journal of School Psychology, Inc.
THE I N F L U E N C E OF TEST SCORES A N D N A T U R A L L Y O C C U R R I N G PUPIL C H A R A C T E R I S T I C S ON PSYCHOEDUCATIONAL DECISION MAKING WITH CHILDREN JAMES YSSELDYKE, BOB ALGOZZINE, RICHARD REGAN, and MATTHEW McGUE University of Minnesota
Summary:A computer simulation of psychoeducational decision making was used to study the extent to which the assessment and decision-making process differs for different kinds of students, the extent to which naturally-occurring pupil characteristics (appearance, sex) influence diagnostic outcomes, and the extent to which decision makers perceive different assessment information and pupil characteristics as influencing their decisions. The decision-making process did not differ as a function of differences in referral information. While SES, sex and physical appearance did not affect outcome decisions, the nature of the reason for referral did. Scores on achievement tests, intelligence tests, and the disparity between the two were rated as having the greatest influence on the eligibility, classification, and prognostic decisions that were made. Decision makers said they were not influenced by naturally-occurring pupil characteristics. School personnel regularly must decide who, among those students experiencing academic and behavioral difficulties, should be declared eligible for and receive special education services. Considerable time and effort go into the collection of data for decision making and in the actual deliberations that lead to decisions. Yet we know very little about the extent to which specific kinds of data influence the decision-making process and its outcomes. Professionals charged with the task of making psychoeducational decisions about students routinely administer or use the results of pupil performance on standardized tests during the decision-making process. Test data are collected to facilitate the making of screening, eligibility/classification/identification/placement,intervention, and evaluation decisions (Salvia & Ysseldyke, 1978). Apparently, test data are collected because someone believes they are important to and useful in decision making. While a number of investigators have reported the frequency with which various kinds of tests are used in practice (Levine, 1974; Silverstein, 1963; Thurlow & Ysseldyke, 1979; Santamafia, Note 1), there are no investigations reporting specifically the kinds of tests used by different practitioners with the same referred students, and few data on the extent to which decision makers perceive different kinds of test information as influencing the decisions they make. Matuszek and OakThe research reported herein was supported by Contract//300-77-0491 between the Bureau of Education for the Handicapped and the University of Minnesota Institute for Research on Learning Disabilities. Special appreciation is extended to Ed Arndt, Martha Bordwell, Patricia Chase, Jean Greener, Joyce Halverson, Martha Thurlow, and Mary Turnblom for assistance in data collection. 167
Journal of School Psychology
168
land (1979) demonstrated that both classroom teachers and school psychologists consider IQ, tested achievement, classroom achievement, and home-related anxiety important in making decisions about pupils, but that psychologists weight IQ and tested achievement more heavily than do teachers. Considerable data do exist demonstrating that both professional-student interpersonal interactions and the assessment process are differentially affected by naturally-occurring pupil characteristics (e.g., race, sex, socioeconomic status, physical attractiveness, etc.). For example, it has been demonstrated that teachers interact differently with black and white students (Coates, 1972; Rubovits & Maehr, 1973) and with girls and boys (Meyer & Thompson, 1956). It has also been reported that pupil's sex differentially affects the kinds of academic and social difficulties decision makers expect students to demonstrate (Algozzine & Ysseldyke, 1980; Schlosser & Aigozzine, 1979). Jackson and Lahaderne (1967) showed that pupil socioeconomic status differentially affects teacher-pupil interactions, while several investigators (Bcrscheid & Walster, 1974; Ross & Salvia, 1975; Algozzine, Note 2) demonstrated that a pupil's physical attractiveness differentially affects both interactions and diagnostic outcomes. This investigation was designed to ascertain (a) the extent to which the assessment process differs (i.e., diagnostic personnel actually use different kinds of assessment information) as a function of differences in referral information on a student; (b) the extent to which different naturally-occurring pupil characteristics influence the outcome decisions made by diagnostic personnel; (c) the extent to which decision makers perceived different kinds of assessment information as influencing their outcome decisions; and (d) the extent to which decision makers perceived naturallyoccurring pupil characteristics as influencing their outcome decisions. The following specific research questions were addressed: 1. 2. 3. 4.
What specific kinds of assessment data were used as a function of referral information (pupil sex, SES, appearance, and type of problem)? To what extent do specific pupil characteristics (pupil sex, SES, physical appearance, type of problem) bias outcome decisions? To what extent do decision makers perceive different kinds of assessment data as influencing their decisions? To what extent do decision makers perceive naturally-occurring characteristics as influencing their decisions? METHOD
Subjects Subjects in a computer-simulated decision-making program were 159 educators and school psychologists in Minnesota. All participants were professionals who had previously participated in at least two placement team meetings. Subjects represented a broad spectrum of disciplines and experience in providing both direct and indirect services in educational settings. Subjects included 22 school psychologists, 44 special education teachers, 52 regular education teachers, 13 administrators, and 13 support personnel (counselors, nurses, social workers, etc.).
Ysseldyke, Algozzine, Regan and McGue
169
Procedure Each of the subjects was asked to read a case folder description of a child and then participate in a diagnostic simulation program developed specifically for this research. The program permitted the subject to access information from an archive which included the results of a variety of assessment devices. Specifically, scores were available for intelligence, achievement, perceptual-motor, personality, and language tests; performance on adaptive behavior scales and the results of several forms of behavioral observation or behavior checklists were also included in the archive. The subject was allowed to select specific tests (e.g., WISC-R, ITPA, etc.) from the seven domains until he/she indicated readiness to make a diagnostic decision; the program then presented a series of decision questions. Regardless of the
specific devices selected, the simulation program consistently provided the participants with data indicating that test performance was within the average range. The entire sequence of activities required approximately 45 minutes to complete a n d each subject was paid $10 for participating. REFERRAL CONDITIONS Prior to receiving the initial case description, each subject was randomly assigned to one o f 16 treatment conditions. The child's sex, socioeconomic status, type of referral problem, and attractiveness was varied to represent each differential condition. The name was listed as Phyllis or William, and the problem was said to be either academic or behavioral in nature. In 8 of the 16 conditions, the referral folder contained information indicating that the student's father was a bank vice president while the mother was a realtor (high SES condition); in the other eight conditions subjects were told that the student's father was a bank janitor, and that the mother was employed as a check-out clerk at a local supermarket (low SES condition). Additionally, previously judged photographs were attached to the case folders to produce an "attractive" or "unattractive" child. Dependent variables. Upon completion of reviewing the case folder and accessing the desired assessment information, each subject completed responses to a series of questions. All were in Likert scale format and asked the participant three diagnostic questions (e.g., to what extent do you believe the referred student is learning disabled?), three prognostic questions (e.g., to what extent do you believe the referred student will have difficulty acquiring math skills?), questions asking them to rate the perceived influence of different kinds of scores (e.g., to what extent did the pupil's scores on intelligence tests influence your decision?), and questions asking them to rate the perceived influence of pupil characteristics (e.g., to what extent did the pupil's sex influence your decision?). The dependent variables were selected to ascertain both the predictions held by decision makers, and factors they thought most heavily influenced their predictions. Data Analysis. The computer simulation program recorded each of the specific tests used by each participant, and the data were treated descriptively. The effects of naturally-occurring pupil characteristics (sex, SES, appearance, and nature of presenting problem) on diagnostic and prognostic decisions were analyzed using two separate four-factor (2 × 2 × 2 × 2)multivariate analyses o f variance. Significant
170
Journal of School Psychology
multivariate effects were subjected to univariate analyses for each dependent variable as appropriate; any further effects were analyzed using t tests. Research questions three and four, investigations of perceived influence of assessment data and naturaily-occurring characteristics on decisions, were addressed by multivariate profile analyses (Morrison, 1976). Results. The simulation program recorded the tests used by each participant. These are listed by referral condition in Table 1. Of the seven domains of data made available, intelligence and achievement tests were selected most frequently, adaptive behavior measures were least often used, and the remaining four domains were selected with similar frequency. The second research question concerned the extent to which specific naturally occurring pupil characteristics biased outcome decisions. Multivariate analyses of variance (MANOVAs) were run separately for the two outcome variables--diagnosis and prognosis. The MANOVA for diagnosis yielded one significant effect; univariate follow-up analyses yielded significant main effects only for the diagnostic decision of emotional disturbance. The case study child was more likely to be rated as disturbed when the presenting problem was behavioral ( X = 3.2) than when it was academic in nature ( X = 3.8). The MANOVA for prognosis yielded one significant effect; univariate follow-up analyses yielded significant main effects only for the prognosis of math difficulty. The child was predicted to have significantly more math difficulties if the reason for referral was academic ( ~ = 2. 6) than if the problem was behavioral (~,T = 2./). Research question three concerned the extent to which decision makers perceived different kinds of assessment data as influencing their outcome decisions. A profile analysis was used to answer this question; sex, SES, appearance, and referral problem were treated as independent variables, with dependent variables being Likert ratings on the perceived usefulness of scores on intelligence, achievement, perceptual-motor, adaptive behavior, personality, and language tests, abilityachievement discrepancies, and subtest discrepancies. Table 2 lists the obtained means for the perceived influence of each of the nine sources of information by referral condition. First, we examined the extent to which there were main effects for types of assessment data perceived as influential as a function of the different kinds of referral information. Results of an ANOVA revealed no main effects for sex, SES, problem statement, or appearance. Second, we examined interactions between the referral information and the specific kinds of data perceived as useful. The Wilk's Lambda for problem statement was significant, and post hoc analysis revealed that data on personality and behavior recordings were more influential in conditions in which the student was referred for behavior problems. The plot of means for the perceived influence of different kinds o f information is reproduced in Figure 1. Inspection reveals that scores on measures of personality and behavior recordings were perceived as having essentially a neutral effect (neither significant nor insignificant effect) when students w e r e referred for academic problems. In those instances where the student was referred for behavioral problems, data from personality tests and behavior recordings were perceived as having greater influence on decision making, though the data still was not perceived as having a significant influence. The ratings were different, though, in the neutral range.
Table 1 Frequency and Proportion of Test Usage as a Function of Referral Condition Domains
Sex 1 2 3 4 5 6 7 8 9 10 I1 12 13 14 15 16 Actual
Male Male Male Male Male Male Male Male Female Female Female Female Female Female Female Female
SES High High High High Low Low Low Low High High High High Low Low Low LOw
Problem Acad. Acad. Bch. Beh. Acad. Acad. Beh. Beh. Acad. Acad. Beh. Beh. Acad. Acad. Beh. Beh.
Attribute Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive
Intelligence
Achievement
Perceptual Motor
Adaptive Behavior Scales
Behavioral Recordings
Language
Personality
Totals
14 15 11 10 15 16 9 16 17 14 13 10 13 15 11 10
19 17 18 13 13 19 15 16 19 20 13 17 19 23 24 21
6 13 5 5 7 10 12 7 12 9 5 8 7 14 5 8
6 7 12 14 7 10 12 9 5 5 15 7 3 4 10 14
4 4 16 10 7 7 8 7 6 5 10 5 3 3 4 10
0 4 3 2 5 2 6 3 5 4 4 3 2 3 3 3
8 5 4 4 6 3 7 4 5 3 5 4 7 6 6 8
57 65 69 58 60 67 69 62 69 60 65 54 54 68 63 74
209
.245 .230 .159 .172 .25* .238 .13 .258* .246 .233 .2 .185 .240 .22 .174 .135
286
.333* .261" .260* .224* .216 .283* .217" .258* .275* .333* .2 .314" .351" .338* .380* .283*
133
.105 .2 .072 .086 .116 .149 .173 .112 .173 .15 .076 .148 .129 .205 .079 .108
.105 .107 .173 .241 .116 .149 .173 .145 .072 .083 .23* .129 .055 .058 .158 .189
140
109
.070 .061 .231 .172 .116 .104 .115 .112 .086 .083 .153 .092 .055 .044 .063 .135
o.
52
.061 .043 .034 .083 .029 .086 .048 .072 .066 .061 .055 .037 .044 .047 .040
85
.140 .076 .057 .068 .10 .044 .101 .064 .072 .05 .076 .074 .129 .088 .095 .108
~" ¢~
~.
O
tJo
o
1014
*most frequently used domain by condition
,...t
172
Journal of School Psychology Table 2 Means for Influence of Assessment Data on Outcome Decisions
Exp Cond Assessment Data
Male
Intelligence Achievement Perceptual-Motor Adaptive Behavior Ability-Achievement Personality Behavioral Recordings Suhtest Discrepancy Language Average
2.06 1.71 2.80 2.89 2.05 2.70 2.23 2.68 3.09 2.47
Sex SES Female High Low
1.97 1.82 2.90 2.96 1.94 3.21 2.61 2.63 3.27 2.59
2.11 1.76 2.75 2.82 1.96 2.97 2.33 2.65 3.16 2.50
1.93 1.78 2.95 3.03 2.03 2.94 2.50 2.66 3.19 2.56
Attractiveness Yes No
2.07 1.83 2.90 2.86 2.06 2.86 2.40 2.78 3.01 2.53
1.96 1,71 2,79 2,99 i,92 3,05 2,44 2.53 3.35 2.53
Problem Acad Beh
Overall Mean
1.80 1.69 2.66 3.06 1.89 3.26 2.69 2.54 3.04 2.51
2.02 1.77 2.85 2.92 1.99 2.96 2.42 2.65 3.18
2.24 1.85 3.04 2.78 2.10 2.65 2.14 2.77 3.32 2.54
Note: 1 = very significant influence, 5 = no influence
A third analysis relative to research question three examined the extent to which there were differences in the specific kinds of data that were perceived as useful across the conditions. The obtained T 2 was significant (p<.00001) indicating that three kinds of data (scores on achievement tests, scores on intelligence tests, and the ability-achievement discrepancy) were perceived as significantly more influential than other kinds of data. The fourth research question concerned the extent to which naturally occurring pupil characteristics were p e r c e i v e d as having an influence on the decisions made. Means for each of the conditions are fisted in Table 3. Profile analysis was used to answer the research question. An analysis of variance was used to examine main effects. A main effect was obtained for SES, and post hoc analysis indicated that SES was rated as having significantly less perceived influence when the student was low SES than when he or she was high SES (X = 4.08 vs. 3.67; t = 2.35, p < .01). Secondly, we examined the extent to which the profiles of means were parallel for the different referral conditions. The Wilk's Lambda for problem statement was significant (p < .04). The means are graphically displayed in Figure 2, and post hoc analysis revealed that the departure from parallelism could be attributed to the general downplay of appearance in the academic condition in contrast to the relatively greater importance attributed to other variables in this condition. Finally, we examined the extent to which the mean ratings for perceived influence of the naturally occurring characteristics differed over all conditions combined. Analysis via the T ~statistic revealed that the means were not equal (p < .00001). Post hoc analysis revealed that problem statement was perceived as significantly more influential than were SES, sex, or attractiveness. SES and attractiveness, further, were perceived as significantly more influential than was sex. DISCUSSION Data collected in assessment should be functionally useful in educational decision making. In this investigation educational decision makers were presented with
Ysseldyke, Algozzine, Regan and McGue
173
Figure 1. Perceived influence of different kinds of data when students were referred for different kinds of problems.
Very
Significant
,.5f
1.6
1.7 Significant
2.0~ 2.5
Neutral
:5.0--
--,4 :5.5
4.0Insignificant
4.4-4.5-m
B e h a v i o r Problem o - - - - o A c a d e m i c Problem
m
Very Insignificant
D
-0
/j/ll///F
If#?
174
Journal of School Psychology Table 3 Means for Influence of Naturally Occurring Pupil Characteristics
Exp Cond Characteristic
Male
Sex Importance SES Importance Appearance Referral Problem Average
4.19 3.73 4.14 1.83 3.47
Sex SES Female High Low 4.43 4.01 4.01 1.86 3.58
4.19 3.67 3.96 1.84 3.42
4.43 4.08 4.19 1.84 3.64
Attractiveness Yes No 4.27 3.90 4.10 1.79 3.52
4.34 3,84 4.05 1,90 3,53
Problem Acad Beh
Overall Means
4.29 3.75 4.22 1.73 3.50
4,31 3.87 4.08 1.84
4.33 4.00 3.92 1.96 3.55
Note: 1 = very significant influence, 5 = no influence
referral information varied only on the sex, socioeconomic status, physical appearance, and type of referral problem. They were given an opportunity to select specific kinds of assessment data, all of which indicated pupil performance and behavior within the average range, were asked to make diagnostic and prognostic decisions, and were asked to report the extent to which specific kinds of test data and naturally occurring pupil information influenced their decisions. Decision makers did perceive naturally-occurring pupil characteristics as socioeconomic status, physical attractiveness, or reason for referral. Rather, across conditions achievement tests were used most often. Referral information affected the outcome decisions made for only one of the four independent variables. The referred student's sex, socioeconomic status, and physical appearance had no effect on the diagnostic and prognostic decisions made. Reason for referral did significantly affect the decision. Although all assessment data indicated average or normal performance, students referred for behavior problems were significantly more often diagnosed and labeled as emotionally disturbed than were students referred for academic problems. Referral information is given considerable weight in making outcome decisions. Decision makers perceive different kinds of assessment data as affecting their outcome decisions. Overall, scores on achievement tests, scores on intelligence tests, and the disparity between the two were perceived as most useful and influential. However, scores on personality tests and behavioral recording data were perceived as having a greater influence on outcome decisions when the referred student demonstrated behavior problems than when he or she demonstrated academic problems. Decision makers did perceive naturally-occurring pupil characteristics as influencing their decisions. Specifically, socioeconomic status was said to influence decisions more when the student was from a high than from a low socioeconomic environment. Secondly, sex, socioeconomic status, and reason for referral were said to have a greater influence on outcome decisions than was physical appearance, but only when the reason for referral was academic in nature. Participants reported that reason for referral has a pronounced effect on outcome decisions, having a significantly greater effect on decisions than did sex, appearance, or socioeconomic status.
175
Ysseldyke, Algozzine, Regan and McGue Figure 2. Perceived Influence of Naturally-Occurring Pupil Characteristics for Academic and Behavior Problems.
Very Significant
1.5
1.6
1.7
b
h
Significant
2.0
B
m
2.5 B
h
/
B
I
Neutral
3.0
b
D
B
3.5 m
B
I
i
4.0
///
B
~"~I
B
R
Insignificant
b
4.4 4.5
B
:
~. Behavior Problem o---..o Academic Problem
I
m
Very
Insignificant
I
I Sex Import
I
I
I
SES Appearance Referral Import Problem
176
Journal of School Psychology
The findings of this study, when considered in the context of several other recent investigations, raise significant concerns about current assessment and decisionmaking efforts in school settings. Thurlow and Ysseldyke (1979) reported that a very large number of tests are used by professionals in making decisions about students, and that there is a generalized failure to differentiate assessment practice as a function of the kind of decision to be made. The findings of this investigation are consistent with those of Matuszek and Oakland (1979) in that decision makers report heavy reliance on scores pupils earn on measures of intelligence and achievement. Yet, this investigation also illustrated the influence of naturally occurring pupil characteristics like SES, physical appearance, and sex. While decision makers use many tests and are influenced heavily by test results, the tests they use are for the most part technically inadequate (Thurlow & Ysseldyke, 1979; Ysseldyke, Algozzine, Regan & Potter, 1980). While decision makers report heavy reliance on test data, it has been demonstrated that there is very little relationship between the decisions reached by placement teams and the actual extent to which the data presented at team meetings support those decisions (Ysseldyke, Algozzine, Richey & Graden, in press). Algozzine and Ysseldyke (in press) found that 51% of a group of over 200 decision makers were willing to declare a psychometrically normal student eligible for special education services. The research reported in this investigation illustrates the tremendous importance which decision makers place on the referral of students for evaluation. Apparently, it is presumed that when teachers refer a student "something" is wrong with the student. Assessment, then, consists of administration of numerous tests in an effort to discover the source of the student's problem. Nationally, there is considerable correspondence between the decision to refer students for evaluation and the decision to declare them eligible for special education services. In New York City during the 1978-79 school year, 95% of those referred for evaluation were declared eligible for special education services. We believe that those who assess students must begin asking very serious questions about the purposes served by assessment. Increased research at the level of referral may help us address some of the assumptions on which we operate. Increased intervention prior to referral, in an effort to short-circuit the relationship between referral and identification as handicapped, is necessary. Assessment must become more than a confirmatory process. REFERENCE NOTES 1. Santamaria, P. J. Psychological testing practices in state residential institutions for the mentally retarded. Unpublished Master's thesis, Pennsylvania State University, 1975. 2. Algozzine, R. F. Attractiveness as a biasing factor in teacher-pupil interactions. Unpublished doctoral dissertation, Pennsylvania State University, 1975. REFERENCES Algozzine, B., & Ysseldyke, J. E. Decision makers' prediction of students' academic difficulties as a function of referral information. Journal of Educational Research, 1980, 73, 145-150. Algozzine, B., & Ysseldyke, J. E. Special education services for normal children: Better safe than sorry? Exceptional Children, in press. Berscheid, E., & Walster, E. Physical attractiveness. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 7). New York: Academic Press, 1974.
Ysseldyke, Algozzine, Regan and M c G u e
177
Coates, B. White adult behavior toward black and white children. Child Development, 1972, 43, 143-154. Jackson, P., & Lahaderne, H. Inequalities of teacher-pupil contacts. Psychology in the Schools, 1967, 4, 204-211. Levine, E. Psychological tests and practices with the deaf: A survey of the state of the art. Volta Review, 1974, 76, 298-319. Matuszek, P., & Oakland, T. Factors influencing teachers' and psychologists' recommendations regarding special class placement. Journal of School Psychology, 1979, 17, 116-125. Meyer, W., & Thompson, G. Sex differences in the distribution of teacher approval and disapproval among sixth grade children. Journal of Educational Psychology, 1956, 47, 385-396. Morrison, D. F. Multivariate statistical methods. New York: McGraw-Hill, 1976. Ross, M. B., & Salvia, J. Attractiveness as a biasing factor in teacher judgments. American Journal of Mental Deficiency, 1975, 80, 96-98. Rubovits, P., & Maehr, M. Pygmalion black and white. Journal of Personality and Social Ps~chology, 1973, 25, 210-218. Salvia, J., & Ysseldyke, J. E. Assessment in special and remedial education. Boston: Houghton-Mifflin, 1978. Schlosser, L., & Algozzine, B. The disturbing child: He or she? The Alberta Journal of Educational Research, 1979, 25, 30-36. Silverstein, A. B. Psychological testing practices in state institutions for the mentally retarded. American Journal of Mental Deficiency, 1963, 68, 440-445. Thurlow, M. L., & Ysseldyke, J. E. Current assessment and decision-making practices in model programs for learning disabled students. Learning Disability Quarterly, 1979, 2, 15-24. Ysseldyke, J. E., Algozzine, B., Regan, R., & Potter, M. Technical adequacy of tests used by professionals in simulated decision making. Psychology in the Schools, 1980, 17, 202-209. Ysseldyke, J. E., Algozzine, B., Richey, L., & Graden, J. Declaring students eligible for learning disability services: Why bother with the data? Learning Disability Quarterly, in press. James Ysseldyke 350 EUiott Hall University of Minnesota-Twin Cities 75 East River Road Minneapolis, MN 55455
Richard Regan Institute for Research on Learning Disabilities University of Minnesota-Twin Cities Minneapolis, MN 55455
Bob Algozzine Associate Professor Special Education University of Florida Galnesville, FL 32601
Matthew McGue Institute for Research on Learning Disabilities University of Minnesota-Twin Cities Minneapolis, MN 55455
Manuscript received: April 1, 1980 Revision received: October 25, 1980
0022-4405/81/1400-0167500.95 ©The Journal of School Psychology, Inc.
THE I N F L U E N C E OF TEST SCORES A N D N A T U R A L L Y O C C U R R I N G PUPIL C H A R A C T E R I S T I C S ON PSYCHOEDUCATIONAL DECISION MAKING WITH CHILDREN JAMES YSSELDYKE, BOB ALGOZZINE, RICHARD REGAN, and MATTHEW McGUE University of Minnesota
Summary:A computer simulation of psychoeducational decision making was used to study the extent to which the assessment and decision-making process differs for different kinds of students, the extent to which naturally-occurring pupil characteristics (appearance, sex) influence diagnostic outcomes, and the extent to which decision makers perceive different assessment information and pupil characteristics as influencing their decisions. The decision-making process did not differ as a function of differences in referral information. While SES, sex and physical appearance did not affect outcome decisions, the nature of the reason for referral did. Scores on achievement tests, intelligence tests, and the disparity between the two were rated as having the greatest influence on the eligibility, classification, and prognostic decisions that were made. Decision makers said they were not influenced by naturally-occurring pupil characteristics. School personnel regularly must decide who, among those students experiencing academic and behavioral difficulties, should be declared eligible for and receive special education services. Considerable time and effort go into the collection of data for decision making and in the actual deliberations that lead to decisions. Yet we know very little about the extent to which specific kinds of data influence the decision-making process and its outcomes. Professionals charged with the task of making psychoeducational decisions about students routinely administer or use the results of pupil performance on standardized tests during the decision-making process. Test data are collected to facilitate the making of screening, eligibility/classification/identification/placement,intervention, and evaluation decisions (Salvia & Ysseldyke, 1978). Apparently, test data are collected because someone believes they are important to and useful in decision making. While a number of investigators have reported the frequency with which various kinds of tests are used in practice (Levine, 1974; Silverstein, 1963; Thurlow & Ysseldyke, 1979; Santamafia, Note 1), there are no investigations reporting specifically the kinds of tests used by different practitioners with the same referred students, and few data on the extent to which decision makers perceive different kinds of test information as influencing the decisions they make. Matuszek and OakThe research reported herein was supported by Contract//300-77-0491 between the Bureau of Education for the Handicapped and the University of Minnesota Institute for Research on Learning Disabilities. Special appreciation is extended to Ed Arndt, Martha Bordwell, Patricia Chase, Jean Greener, Joyce Halverson, Martha Thurlow, and Mary Turnblom for assistance in data collection. 167
Journal of School Psychology
168
land (1979) demonstrated that both classroom teachers and school psychologists consider IQ, tested achievement, classroom achievement, and home-related anxiety important in making decisions about pupils, but that psychologists weight IQ and tested achievement more heavily than do teachers. Considerable data do exist demonstrating that both professional-student interpersonal interactions and the assessment process are differentially affected by naturally-occurring pupil characteristics (e.g., race, sex, socioeconomic status, physical attractiveness, etc.). For example, it has been demonstrated that teachers interact differently with black and white students (Coates, 1972; Rubovits & Maehr, 1973) and with girls and boys (Meyer & Thompson, 1956). It has also been reported that pupil's sex differentially affects the kinds of academic and social difficulties decision makers expect students to demonstrate (Algozzine & Ysseldyke, 1980; Schlosser & Aigozzine, 1979). Jackson and Lahaderne (1967) showed that pupil socioeconomic status differentially affects teacher-pupil interactions, while several investigators (Bcrscheid & Walster, 1974; Ross & Salvia, 1975; Algozzine, Note 2) demonstrated that a pupil's physical attractiveness differentially affects both interactions and diagnostic outcomes. This investigation was designed to ascertain (a) the extent to which the assessment process differs (i.e., diagnostic personnel actually use different kinds of assessment information) as a function of differences in referral information on a student; (b) the extent to which different naturally-occurring pupil characteristics influence the outcome decisions made by diagnostic personnel; (c) the extent to which decision makers perceived different kinds of assessment information as influencing their outcome decisions; and (d) the extent to which decision makers perceived naturallyoccurring pupil characteristics as influencing their outcome decisions. The following specific research questions were addressed: 1. 2. 3. 4.
What specific kinds of assessment data were used as a function of referral information (pupil sex, SES, appearance, and type of problem)? To what extent do specific pupil characteristics (pupil sex, SES, physical appearance, type of problem) bias outcome decisions? To what extent do decision makers perceive different kinds of assessment data as influencing their decisions? To what extent do decision makers perceive naturally-occurring characteristics as influencing their decisions? METHOD
Subjects Subjects in a computer-simulated decision-making program were 159 educators and school psychologists in Minnesota. All participants were professionals who had previously participated in at least two placement team meetings. Subjects represented a broad spectrum of disciplines and experience in providing both direct and indirect services in educational settings. Subjects included 22 school psychologists, 44 special education teachers, 52 regular education teachers, 13 administrators, and 13 support personnel (counselors, nurses, social workers, etc.).
Ysseldyke, Algozzine, Regan and McGue
169
Procedure Each of the subjects was asked to read a case folder description of a child and then participate in a diagnostic simulation program developed specifically for this research. The program permitted the subject to access information from an archive which included the results of a variety of assessment devices. Specifically, scores were available for intelligence, achievement, perceptual-motor, personality, and language tests; performance on adaptive behavior scales and the results of several forms of behavioral observation or behavior checklists were also included in the archive. The subject was allowed to select specific tests (e.g., WISC-R, ITPA, etc.) from the seven domains until he/she indicated readiness to make a diagnostic decision; the program then presented a series of decision questions. Regardless of the
specific devices selected, the simulation program consistently provided the participants with data indicating that test performance was within the average range. The entire sequence of activities required approximately 45 minutes to complete a n d each subject was paid $10 for participating. REFERRAL CONDITIONS Prior to receiving the initial case description, each subject was randomly assigned to one o f 16 treatment conditions. The child's sex, socioeconomic status, type of referral problem, and attractiveness was varied to represent each differential condition. The name was listed as Phyllis or William, and the problem was said to be either academic or behavioral in nature. In 8 of the 16 conditions, the referral folder contained information indicating that the student's father was a bank vice president while the mother was a realtor (high SES condition); in the other eight conditions subjects were told that the student's father was a bank janitor, and that the mother was employed as a check-out clerk at a local supermarket (low SES condition). Additionally, previously judged photographs were attached to the case folders to produce an "attractive" or "unattractive" child. Dependent variables. Upon completion of reviewing the case folder and accessing the desired assessment information, each subject completed responses to a series of questions. All were in Likert scale format and asked the participant three diagnostic questions (e.g., to what extent do you believe the referred student is learning disabled?), three prognostic questions (e.g., to what extent do you believe the referred student will have difficulty acquiring math skills?), questions asking them to rate the perceived influence of different kinds of scores (e.g., to what extent did the pupil's scores on intelligence tests influence your decision?), and questions asking them to rate the perceived influence of pupil characteristics (e.g., to what extent did the pupil's sex influence your decision?). The dependent variables were selected to ascertain both the predictions held by decision makers, and factors they thought most heavily influenced their predictions. Data Analysis. The computer simulation program recorded each of the specific tests used by each participant, and the data were treated descriptively. The effects of naturally-occurring pupil characteristics (sex, SES, appearance, and nature of presenting problem) on diagnostic and prognostic decisions were analyzed using two separate four-factor (2 × 2 × 2 × 2)multivariate analyses o f variance. Significant
170
Journal of School Psychology
multivariate effects were subjected to univariate analyses for each dependent variable as appropriate; any further effects were analyzed using t tests. Research questions three and four, investigations of perceived influence of assessment data and naturaily-occurring characteristics on decisions, were addressed by multivariate profile analyses (Morrison, 1976). Results. The simulation program recorded the tests used by each participant. These are listed by referral condition in Table 1. Of the seven domains of data made available, intelligence and achievement tests were selected most frequently, adaptive behavior measures were least often used, and the remaining four domains were selected with similar frequency. The second research question concerned the extent to which specific naturally occurring pupil characteristics biased outcome decisions. Multivariate analyses of variance (MANOVAs) were run separately for the two outcome variables--diagnosis and prognosis. The MANOVA for diagnosis yielded one significant effect; univariate follow-up analyses yielded significant main effects only for the diagnostic decision of emotional disturbance. The case study child was more likely to be rated as disturbed when the presenting problem was behavioral ( X = 3.2) than when it was academic in nature ( X = 3.8). The MANOVA for prognosis yielded one significant effect; univariate follow-up analyses yielded significant main effects only for the prognosis of math difficulty. The child was predicted to have significantly more math difficulties if the reason for referral was academic ( ~ = 2. 6) than if the problem was behavioral (~,T = 2./). Research question three concerned the extent to which decision makers perceived different kinds of assessment data as influencing their outcome decisions. A profile analysis was used to answer this question; sex, SES, appearance, and referral problem were treated as independent variables, with dependent variables being Likert ratings on the perceived usefulness of scores on intelligence, achievement, perceptual-motor, adaptive behavior, personality, and language tests, abilityachievement discrepancies, and subtest discrepancies. Table 2 lists the obtained means for the perceived influence of each of the nine sources of information by referral condition. First, we examined the extent to which there were main effects for types of assessment data perceived as influential as a function of the different kinds of referral information. Results of an ANOVA revealed no main effects for sex, SES, problem statement, or appearance. Second, we examined interactions between the referral information and the specific kinds of data perceived as useful. The Wilk's Lambda for problem statement was significant, and post hoc analysis revealed that data on personality and behavior recordings were more influential in conditions in which the student was referred for behavior problems. The plot of means for the perceived influence of different kinds o f information is reproduced in Figure 1. Inspection reveals that scores on measures of personality and behavior recordings were perceived as having essentially a neutral effect (neither significant nor insignificant effect) when students w e r e referred for academic problems. In those instances where the student was referred for behavioral problems, data from personality tests and behavior recordings were perceived as having greater influence on decision making, though the data still was not perceived as having a significant influence. The ratings were different, though, in the neutral range.
Table 1 Frequency and Proportion of Test Usage as a Function of Referral Condition Domains
Sex 1 2 3 4 5 6 7 8 9 10 I1 12 13 14 15 16 Actual
Male Male Male Male Male Male Male Male Female Female Female Female Female Female Female Female
SES High High High High Low Low Low Low High High High High Low Low Low LOw
Problem Acad. Acad. Bch. Beh. Acad. Acad. Beh. Beh. Acad. Acad. Beh. Beh. Acad. Acad. Beh. Beh.
Attribute Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive Attractive Unattractive
Intelligence
Achievement
Perceptual Motor
Adaptive Behavior Scales
Behavioral Recordings
Language
Personality
Totals
14 15 11 10 15 16 9 16 17 14 13 10 13 15 11 10
19 17 18 13 13 19 15 16 19 20 13 17 19 23 24 21
6 13 5 5 7 10 12 7 12 9 5 8 7 14 5 8
6 7 12 14 7 10 12 9 5 5 15 7 3 4 10 14
4 4 16 10 7 7 8 7 6 5 10 5 3 3 4 10
0 4 3 2 5 2 6 3 5 4 4 3 2 3 3 3
8 5 4 4 6 3 7 4 5 3 5 4 7 6 6 8
57 65 69 58 60 67 69 62 69 60 65 54 54 68 63 74
209
.245 .230 .159 .172 .25* .238 .13 .258* .246 .233 .2 .185 .240 .22 .174 .135
286
.333* .261" .260* .224* .216 .283* .217" .258* .275* .333* .2 .314" .351" .338* .380* .283*
133
.105 .2 .072 .086 .116 .149 .173 .112 .173 .15 .076 .148 .129 .205 .079 .108
.105 .107 .173 .241 .116 .149 .173 .145 .072 .083 .23* .129 .055 .058 .158 .189
140
109
.070 .061 .231 .172 .116 .104 .115 .112 .086 .083 .153 .092 .055 .044 .063 .135
o.
52
.061 .043 .034 .083 .029 .086 .048 .072 .066 .061 .055 .037 .044 .047 .040
85
.140 .076 .057 .068 .10 .044 .101 .064 .072 .05 .076 .074 .129 .088 .095 .108
~" ¢~
~.
O
tJo
o
1014
*most frequently used domain by condition
,...t
172
Journal of School Psychology Table 2 Means for Influence of Assessment Data on Outcome Decisions
Exp Cond Assessment Data
Male
Intelligence Achievement Perceptual-Motor Adaptive Behavior Ability-Achievement Personality Behavioral Recordings Suhtest Discrepancy Language Average
2.06 1.71 2.80 2.89 2.05 2.70 2.23 2.68 3.09 2.47
Sex SES Female High Low
1.97 1.82 2.90 2.96 1.94 3.21 2.61 2.63 3.27 2.59
2.11 1.76 2.75 2.82 1.96 2.97 2.33 2.65 3.16 2.50
1.93 1.78 2.95 3.03 2.03 2.94 2.50 2.66 3.19 2.56
Attractiveness Yes No
2.07 1.83 2.90 2.86 2.06 2.86 2.40 2.78 3.01 2.53
1.96 1,71 2,79 2,99 i,92 3,05 2,44 2.53 3.35 2.53
Problem Acad Beh
Overall Mean
1.80 1.69 2.66 3.06 1.89 3.26 2.69 2.54 3.04 2.51
2.02 1.77 2.85 2.92 1.99 2.96 2.42 2.65 3.18
2.24 1.85 3.04 2.78 2.10 2.65 2.14 2.77 3.32 2.54
Note: 1 = very significant influence, 5 = no influence
A third analysis relative to research question three examined the extent to which there were differences in the specific kinds of data that were perceived as useful across the conditions. The obtained T 2 was significant (p<.00001) indicating that three kinds of data (scores on achievement tests, scores on intelligence tests, and the ability-achievement discrepancy) were perceived as significantly more influential than other kinds of data. The fourth research question concerned the extent to which naturally occurring pupil characteristics were p e r c e i v e d as having an influence on the decisions made. Means for each of the conditions are fisted in Table 3. Profile analysis was used to answer the research question. An analysis of variance was used to examine main effects. A main effect was obtained for SES, and post hoc analysis indicated that SES was rated as having significantly less perceived influence when the student was low SES than when he or she was high SES (X = 4.08 vs. 3.67; t = 2.35, p < .01). Secondly, we examined the extent to which the profiles of means were parallel for the different referral conditions. The Wilk's Lambda for problem statement was significant (p < .04). The means are graphically displayed in Figure 2, and post hoc analysis revealed that the departure from parallelism could be attributed to the general downplay of appearance in the academic condition in contrast to the relatively greater importance attributed to other variables in this condition. Finally, we examined the extent to which the mean ratings for perceived influence of the naturally occurring characteristics differed over all conditions combined. Analysis via the T ~statistic revealed that the means were not equal (p < .00001). Post hoc analysis revealed that problem statement was perceived as significantly more influential than were SES, sex, or attractiveness. SES and attractiveness, further, were perceived as significantly more influential than was sex. DISCUSSION Data collected in assessment should be functionally useful in educational decision making. In this investigation educational decision makers were presented with
Ysseldyke, Algozzine, Regan and McGue
173
Figure 1. Perceived influence of different kinds of data when students were referred for different kinds of problems.
Very
Significant
,.5f
1.6
1.7 Significant
2.0~ 2.5
Neutral
:5.0--
--,4 :5.5
4.0Insignificant
4.4-4.5-m
B e h a v i o r Problem o - - - - o A c a d e m i c Problem
m
Very Insignificant
D
-0
/j/ll///F
If#?
174
Journal of School Psychology Table 3 Means for Influence of Naturally Occurring Pupil Characteristics
Exp Cond Characteristic
Male
Sex Importance SES Importance Appearance Referral Problem Average
4.19 3.73 4.14 1.83 3.47
Sex SES Female High Low 4.43 4.01 4.01 1.86 3.58
4.19 3.67 3.96 1.84 3.42
4.43 4.08 4.19 1.84 3.64
Attractiveness Yes No 4.27 3.90 4.10 1.79 3.52
4.34 3,84 4.05 1,90 3,53
Problem Acad Beh
Overall Means
4.29 3.75 4.22 1.73 3.50
4,31 3.87 4.08 1.84
4.33 4.00 3.92 1.96 3.55
Note: 1 = very significant influence, 5 = no influence
referral information varied only on the sex, socioeconomic status, physical appearance, and type of referral problem. They were given an opportunity to select specific kinds of assessment data, all of which indicated pupil performance and behavior within the average range, were asked to make diagnostic and prognostic decisions, and were asked to report the extent to which specific kinds of test data and naturally occurring pupil information influenced their decisions. Decision makers did perceive naturally-occurring pupil characteristics as socioeconomic status, physical attractiveness, or reason for referral. Rather, across conditions achievement tests were used most often. Referral information affected the outcome decisions made for only one of the four independent variables. The referred student's sex, socioeconomic status, and physical appearance had no effect on the diagnostic and prognostic decisions made. Reason for referral did significantly affect the decision. Although all assessment data indicated average or normal performance, students referred for behavior problems were significantly more often diagnosed and labeled as emotionally disturbed than were students referred for academic problems. Referral information is given considerable weight in making outcome decisions. Decision makers perceive different kinds of assessment data as affecting their outcome decisions. Overall, scores on achievement tests, scores on intelligence tests, and the disparity between the two were perceived as most useful and influential. However, scores on personality tests and behavioral recording data were perceived as having a greater influence on outcome decisions when the referred student demonstrated behavior problems than when he or she demonstrated academic problems. Decision makers did perceive naturally-occurring pupil characteristics as influencing their decisions. Specifically, socioeconomic status was said to influence decisions more when the student was from a high than from a low socioeconomic environment. Secondly, sex, socioeconomic status, and reason for referral were said to have a greater influence on outcome decisions than was physical appearance, but only when the reason for referral was academic in nature. Participants reported that reason for referral has a pronounced effect on outcome decisions, having a significantly greater effect on decisions than did sex, appearance, or socioeconomic status.
175
Ysseldyke, Algozzine, Regan and McGue Figure 2. Perceived Influence of Naturally-Occurring Pupil Characteristics for Academic and Behavior Problems.
Very Significant
1.5
1.6
1.7
b
h
Significant
2.0
B
m
2.5 B
h
/
B
I
Neutral
3.0
b
D
B
3.5 m
B
I
i
4.0
///
B
~"~I
B
R
Insignificant
b
4.4 4.5
B
:
~. Behavior Problem o---..o Academic Problem
I
m
Very
Insignificant
I
I Sex Import
I
I
I
SES Appearance Referral Import Problem
176
Journal of School Psychology
The findings of this study, when considered in the context of several other recent investigations, raise significant concerns about current assessment and decisionmaking efforts in school settings. Thurlow and Ysseldyke (1979) reported that a very large number of tests are used by professionals in making decisions about students, and that there is a generalized failure to differentiate assessment practice as a function of the kind of decision to be made. The findings of this investigation are consistent with those of Matuszek and Oakland (1979) in that decision makers report heavy reliance on scores pupils earn on measures of intelligence and achievement. Yet, this investigation also illustrated the influence of naturally occurring pupil characteristics like SES, physical appearance, and sex. While decision makers use many tests and are influenced heavily by test results, the tests they use are for the most part technically inadequate (Thurlow & Ysseldyke, 1979; Ysseldyke, Algozzine, Regan & Potter, 1980). While decision makers report heavy reliance on test data, it has been demonstrated that there is very little relationship between the decisions reached by placement teams and the actual extent to which the data presented at team meetings support those decisions (Ysseldyke, Algozzine, Richey & Graden, in press). Algozzine and Ysseldyke (in press) found that 51% of a group of over 200 decision makers were willing to declare a psychometrically normal student eligible for special education services. The research reported in this investigation illustrates the tremendous importance which decision makers place on the referral of students for evaluation. Apparently, it is presumed that when teachers refer a student "something" is wrong with the student. Assessment, then, consists of administration of numerous tests in an effort to discover the source of the student's problem. Nationally, there is considerable correspondence between the decision to refer students for evaluation and the decision to declare them eligible for special education services. In New York City during the 1978-79 school year, 95% of those referred for evaluation were declared eligible for special education services. We believe that those who assess students must begin asking very serious questions about the purposes served by assessment. Increased research at the level of referral may help us address some of the assumptions on which we operate. Increased intervention prior to referral, in an effort to short-circuit the relationship between referral and identification as handicapped, is necessary. Assessment must become more than a confirmatory process. REFERENCE NOTES 1. Santamaria, P. J. Psychological testing practices in state residential institutions for the mentally retarded. Unpublished Master's thesis, Pennsylvania State University, 1975. 2. Algozzine, R. F. Attractiveness as a biasing factor in teacher-pupil interactions. Unpublished doctoral dissertation, Pennsylvania State University, 1975. REFERENCES Algozzine, B., & Ysseldyke, J. E. Decision makers' prediction of students' academic difficulties as a function of referral information. Journal of Educational Research, 1980, 73, 145-150. Algozzine, B., & Ysseldyke, J. E. Special education services for normal children: Better safe than sorry? Exceptional Children, in press. Berscheid, E., & Walster, E. Physical attractiveness. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 7). New York: Academic Press, 1974.
Ysseldyke, Algozzine, Regan and M c G u e
177
Coates, B. White adult behavior toward black and white children. Child Development, 1972, 43, 143-154. Jackson, P., & Lahaderne, H. Inequalities of teacher-pupil contacts. Psychology in the Schools, 1967, 4, 204-211. Levine, E. Psychological tests and practices with the deaf: A survey of the state of the art. Volta Review, 1974, 76, 298-319. Matuszek, P., & Oakland, T. Factors influencing teachers' and psychologists' recommendations regarding special class placement. Journal of School Psychology, 1979, 17, 116-125. Meyer, W., & Thompson, G. Sex differences in the distribution of teacher approval and disapproval among sixth grade children. Journal of Educational Psychology, 1956, 47, 385-396. Morrison, D. F. Multivariate statistical methods. New York: McGraw-Hill, 1976. Ross, M. B., & Salvia, J. Attractiveness as a biasing factor in teacher judgments. American Journal of Mental Deficiency, 1975, 80, 96-98. Rubovits, P., & Maehr, M. Pygmalion black and white. Journal of Personality and Social Ps~chology, 1973, 25, 210-218. Salvia, J., & Ysseldyke, J. E. Assessment in special and remedial education. Boston: Houghton-Mifflin, 1978. Schlosser, L., & Algozzine, B. The disturbing child: He or she? The Alberta Journal of Educational Research, 1979, 25, 30-36. Silverstein, A. B. Psychological testing practices in state institutions for the mentally retarded. American Journal of Mental Deficiency, 1963, 68, 440-445. Thurlow, M. L., & Ysseldyke, J. E. Current assessment and decision-making practices in model programs for learning disabled students. Learning Disability Quarterly, 1979, 2, 15-24. Ysseldyke, J. E., Algozzine, B., Regan, R., & Potter, M. Technical adequacy of tests used by professionals in simulated decision making. Psychology in the Schools, 1980, 17, 202-209. Ysseldyke, J. E., Algozzine, B., Richey, L., & Graden, J. Declaring students eligible for learning disability services: Why bother with the data? Learning Disability Quarterly, in press. James Ysseldyke 350 EUiott Hall University of Minnesota-Twin Cities 75 East River Road Minneapolis, MN 55455
Richard Regan Institute for Research on Learning Disabilities University of Minnesota-Twin Cities Minneapolis, MN 55455
Bob Algozzine Associate Professor Special Education University of Florida Galnesville, FL 32601
Matthew McGue Institute for Research on Learning Disabilities University of Minnesota-Twin Cities Minneapolis, MN 55455
Manuscript received: April 1, 1980 Revision received: October 25, 1980