- Email: [email protected]
Decision analysis: An approach for multidisciplinary teams in planning special service programs
Journal of School Psychology 1981=Vol. 19, No. 4
0022-4405/81/1600-0340500.95 © 1981 The Journal of School Psychology, Inc.
D E C I S I O N A N A L Y S I S : A N A P P R O A C H FOR M U L T I D I S C I P L I N A R Y T E A M S IN P L A N N I N G S P E C I A L SERVICE P R O G R A M S CHARLES A. MAHER Rutgers University
Summary: A decision analytic approach for use by multidisciplinary teams (MDTs) in planning special service programs is described, and formative evaluation information on its application by MDTs in planning individualized education programs (IEPs) is presented. Decision analysis is applied following initial determination of program goals and objectives. In~using the approach, an MDT engages in a six-step process: (a) delineation of program alternatives, (b) determination of program outcomes, (c) assessment of probabilities, (d) assessment of utilities, (e) determination of overall program values, and (0 selection of the program to be developed. Utilization of the approach allows a complex program planning situation to be disaggregated into its essential elements, with discussion of program alternatives occurring relative to specific decision criteria. Strengths and limitations of the approach also are discussed, and future directions for research are briefly noted. The planning of special service programs by multidisciplinary teams (MDTs) is a complex decision making endeavor (Yoshida, Fcaton, Maxwell, & Kaufman, 1978). Whether the program to be planned is an individualized education program (IEP), a resource room, or an alternate school, it is unlikely that an MDT will have complete understanding of all decisions to be made or accurate knowledge of all future events likely to affect its planning decisions (Maher & Barbrack, 1979; Mitroff, Emshoff, & Kilmann, 1979). The complexity and uncertainty of special service program planning can be seen by examining the kinds of decision questions an MDT might consider when planning an IEP for a handicapped child: On what goals and objectives will the IEP focus? What alternative programs may be developed? What positive and negative effects might accrue from each alternative? Which alternative is seen as more worthwhile by professionals, parents, child? Analysis and synthesis of information for an informed program choice is not a straightforward matter for an MDT. Typically, integration of decision making information occurs in a tacit fashion, largely because the team does not possess a framework for disaggregating a global planning decision into its essential elements. Although the art of decision making should not be minimized, decision making units such as MDTs may improve their planning ability by means of decision analysis, a planning procedure derived from decision theory (see Keeney & Ralffa, 1976). In this paper, a decision analytic procedure for use by MDTs in planning special service programs is described, formative evaluation information to support the potential utility of the approach is presented, strengths and limitations of the approach are discussed, and directions for future research are noted. 340
Maher
341
THE DECISION-ANALYTIC APPROACH Decision analysis is described and illustrated below as a six-step approach: (a) delineation of program alternatives, (b) determination of program outcomes, (c) assessment of probabilities, (d) assessment of utilities, (e) determination of overall program values, and (f) selection of theiprogram to be developed. The approach is used by an MDT following identification and assessment of a program client (i.e., individual pupil, group of pupils) and the setting of goals and objectives for that client. For IEPs, decision analysis is applied subsequent to the formulation of IEP goals and objectives by the team, but prior to the placement decision. For' group programs (e.g., resource rooms), the approach is used following discussion of general goals (procedure manual is available for author). The approach has been determined by state department of education officials as being congruent with the IEP provisions of PL 94-142.
Delineation o f Program Alternatives In this step, the MDT, based upon previously established goals and objectives, delineates the program alternatives to be considered, portraying each option by means of a decision tree (see Figure 1). A decision tree provides clarity to the program planning situation: Besides depicting program alternatives, a decision tree displays the level of outcome expected to result from each alternative and the probabilities and utilities associated with each alternative. A decision tree is composed of a series of nodes and branches. Square nodes represent events that are dictated by the MDT, while round nodes are dictated by chance. In using the decision analytic approach, the sum of the probabilities assigned to each outcome for each program alternative must equal one (see Figure 1). Illustration. A fourth-grade child in a regular education classroom had been identified by the MDT as being eligible to receive special education and related services. The assessment process revealed three areas of need: reading, language arts, and behavioral self-control; and one area of strength: grade-level performance in mathematics. IEP goals and objectives were set in reading, spelling, grammar, and completion of class assignments. Using decision analysis, the MDT, in concert with the child's parents, proposed the two individualized education program alternatives seen in Figure 1: (a) special classroom program, with mainstreaming into music, art, and physical e~ucation; and (b) regular classroom program, with academic instruction in a resource room as well as behavioral self-control education program to be provided by the school counselor.
Determination of Program Outcomes In this step, expected outcomes for each program alternative are determined. Outcomes are observable, measurable criteria, such as IEP objectives, or other indicators that are developed if current goals and objectives are not considered as being amenable to an objective assessment by outside raters. In determining program outcomes, an MDT must consider program risks and benefits. In this regard, a program may improve a client's performance (benefit) or result in negative side effects (risks). For example, while a program for a group of gifted children may be
342
Journal of School Psychology
beneficial in terms of achievement of academic outcomes, the program may be seen as~placing considerable stress on the children whereby negative side effects such as cheating, psychosomatic complaints, or refusal to attend school may develop. Benefits (positive outcomes) of a program alternative can be measured in terms of prevention or (emediation of social or emotional problems, or with respect to acquisition or development of behavioral or academic skills. Risks (negative side effects) resulting from a program can be measured in terms of variables such as increased frustration, refusal to attend school, or academic failure. Illustration. The MDT further operationalized the child's IEP objectives as program outcomes by means of goal attainment scaling, a procedure which specifies levels of outcome (see Maher & Barbrack, 1979). The team decided that a "satisfactory" outcome would be attainment of 75°70 or more of all IEP objectives, at the time of IEP annual review; a "not satisfactory" outcome was considered to be less than 75070 attainment of all goals. Program benefits were considered to be achievement of satisfactory outcomes, while negative effects were seen as less than satisfactory outcomes as well as any maladaptive classroom behavior the child may exhibit.
Assessment o f Probabilities In this step, the probability that each program alternative will result in a particular program outcome is assessed by an MDT. Decision analysis requires greater attention and precision in the assessment of probabilities than usually is the case with MDTs. Often, in special service program planning situations, team members speak of positive program outcomes using terms such as frequent, satisfactory, or high. For each of these qualitative likelihood descriptions, however, a team can set a probability at 1 in 10, another at 1 in 100, or 1 in 1,000. The same kinds of probabilities can be set for negative outcome" descriptions such as uncommon, infrequent, not satisfactory, or low. To the extent possible in assessment of probabilities by an MDT, empirical evidence derived from case studies and more controlled types of program evaluations can serve as the basis for determining the probability of success of program alternatives. Experience dictates, however, that such objective data l ~ y not always be available and, when available, may not be applicable to the program client of interest. Many factors such as the age of the client, the presence or absence of particular psychoeducational strengths and needs, and previous learning experiences may need to be considered in determining probabilities of program alternatives for a particular client. Thus, in decision analysis, probability values are derived from subjective estimates of team members. This can be accomplished by averaging the estimates of individual probability values of team members, or by means of team consensus on a particular value following discussion. If team members disagree on a particular value, they can first discuss their differences and arrive at a compromise value. If disagreement persists, the matter can be referred to another party (e.g., director of special services) for conflict resolution. Procedures for obtaining team consensus on probability values and utility values (to be discussed below) can be found in Butler (Note 1), Delbecq, Van de Ven, and Gustafson (1975), and Keeney and Raiffa (1976). Illustration. The MDT assessed the likelihood of occurrence of the outcomes relating to each program alternative and obtained consensus on those likelihoods
343
Maher PROGRAM ALTERNATIVES
LEVEL OF GOAL A'rrAINMEUT (OUTCOME)
UTILITIES
SATISFACTORY(75%or more) ~ probability= .80
40
OVERALL VALUE
.80 (40) = 32
SPECIALCLASSPROGRAM NOTSATISFACTORY(less than 75%) probability .20
3o
=
I
SATISFACTORY(750 or more)
80
.20 (3o) --s o.v. *= as
.40 (80) = 32
REGULARCLASSPROGRAM(~ probability= .40
| NOT~SA?ISFACTORY : .60
(Less than
probability
75%)
.60 (so) = 36 60
O.V.* = 6g
*O.V. = ~ (Probability of outcome X utility of outcome)
Figure 1. A decision tree for a planning situation involving an IEP.
(see Figure 1). Assessment of probabilities was based not only on historical data (e.g., previous standardized and criterion referenced test scores), but also on the judgments of team members about the child which were derived from their experiences with similar kinds of children they had encountered in the past. Thus, previous learning history of the child, parental opinions, adequacy of the school district's resources, and other factors influenced team members' judgments about the likelihood of each possible outcome.
Assessment of Utilities In this step, an MDT assessess the utility of each program alternative. Assessment of utilities, however, is more complex than that of probabilities, since utilities are indices of value, and judgments of value are essentially subjective in nature. As noted above, various benefits and risks can result from different program alternatives. For a program client, benefits may include a certain level of academic achievement; risks may include potential physical or emotional discomfort, loss of self-esteem, or frustration resulting from inappropriate task demands. In any given program planning situation, several benefits and risks can be anticipated by an MDT as a consequence of a particular program alternative. Thus, it is important for team members to be able to compare the utilities of a number of benefits and risks with differing characteristics. For example, for a particular handicapped child, it may be necessary to compare the potential academic achievement (benefit) attained by the child by means of a special classroom program with the potential ridicule (risks) the
344
Journal of School Psychology
child might receive from nonhandicapped peers and the effect of such derision on the handicapped child's self-esteem. When program outcomes can be given a utility measurement in a single unit such as monetary cost, unidimensional utilities can be used, and they are readily ordered and compared. However, special service program outcomes are multidimensional. Since program outcomes such as academic achievement, behavioral self-control, and physical well-being are measured in different units, comparison and ordering of them by an MDT requires that a common multidimensional utility be devised to express all of them. For example, in planning an alternative school program for a group of socially maladjusted adolescents, if dropping out of school were given a utility value of 0 units, a particular rate of school attendance might be assigned 60 units, while a certain level of academic achievement might be given a utility value of 100 units. Though assessment of probabilities usually is the exclusive responsibility of the team, assessment of utilities often is undertaken in concert with the child and the child's parents. Since different persons assess the value of outcomes differently, it is essential when assigning utilities to agree upon rules for such assignment. For example, the rule for assignment of utilities may im/olve computing the mean of individual team members' utilities, or the rule may involve team discussion and consensus on a compromise utility. Illustration. In this example, even though the probability of satisfactory level of goal attainment for the special classroom program was .80, it had limited value to some team members due to their concern that the child would not be mainstreamed into regular academic classrooms. Moreover, although the probability of satisfactory level of goal attainment in the regular classroom was .40, the utility was seen as high by all team members since the child would be more involved in the mainstream of public school education (Figure 1). From this case example, it is apparent that the~relative worth of each program alternative is a function of both the probability of the outcome of the alternative and the utility of the outcome. Thus, in order to obtain an index of the value of each outcome, it was necessary for the team to develop a scale for assessment of the utility of each outcome. In this example, an arbitrary 100 point scale of relative units was developed by the team. On this scale, a satisfactory level of goal attainment in the special classroom program had a utility of 40 units, while a satisfactory level of goal attainment for the regular classroom program had a utility of 80 units, even though the probability for satisfactory goal attainment was higher in the special classroom program (Figure 1).
Determination o f Overall Program Values Having assigned both probabilities and utilities for each of the outcomes, the next step for a team is to calculate the overall value of each outcome for each program alternative. The value of each outcome is calculated by multiplying the probability of the outcome and the corresponding utility of that outcome. When the values of both branches are calculated, the sum of these values represents the overall expected value for that branch. Illustration. In this case, the overall value for the special class program was 38, since 40 (the utility) times .80 (the probability) equaled 32 (see last column in Figure 1), and 30 (utility) times .20 (probability) equaled 6 (see last column in Figure 1). Thus, the overall value of 32 plus 6 equaled 38 (see last column in Figure 1).
Maher
345
Selection of the Program to be Developed This step involves selection of the " b e s t " program alternative, that is, the alternative with the highest expected overall value. In using decision analysis, opinion about the program selected also can be sought from those not involved in the process (e.g., administrators), especially when it is not clear that a particular alternative is the " b e s t " one (e.g., when ~a program with a low probability has been choosen). If outside individuals or groups also choose the same alternative, the team can be more certain that their alternative is acceptable. If consensus is not obtained, a joint committee could attempt to resolve differences. Illustration. In this case example, the overall value of the regular education program was 68, signifying that the best choice was the regular classroom option. Consultation with parents and with special education and regular education teachers also verified this choice.
Planning Group Educational Programs Decision analysis also can be used by ~n MDT in planning a group program; one such example is briefly summarized here, without presentation of the actual data: Ten high school pupils had been identified by an MDT as being in need of special education services, focusing on a program goal of social adjustment. Using decision analysis, three group program alternatives were considered by the team in relation to the overall goal: (a) a self-contained special education classroom program; (b) a career-oriented program in the regular high school, emphasizing vocational skill development training; and (c) a regular high school program, with remedial instruction in needed areas. The MDT then determined specific outcomes having to do with level of attainment of social goals (e.g., improved peer relationships), assigning numerical probabilities and utilities to each outcome, for each alternative. Subsequent calculations revealed: (a) the regular classroom alternative with the vocational skill development training, although having a lower probability for goal attainment than did the special class alternative, had a much higher utility value. (b) Although the self-contained class program alternative had a high probability with respect to satisfactory goal attainment, the utility of this alternative was relatively low since team members believed that such a program would result in school truancy and lowered self-esteem. (c) The regular classroom alternative with remedial instruction was [een as being low both from probability and utility perspectives. Further calculation of the overall value of each alternative revealed the careeroriented program as the best program choice. USE OF DECISION ANALYSIS BY MDTs: A FORMATIVE EVALUATION A formative evaluation, offering initial exploratory evidence of the potential usefulness of decision analysis, is presented here: An urban public school district in New Jersey employed two full-time MDTs that had worked together for three years: (a) an Elementary School Team, responsible for planning special services in the district's four elementary schools; and (b) a Secondary School Team, with similar responsibilities in the district's middle and high school. Each team consisted of a "core team" of school psychologist, learning consultant, and social worker, with the school principal, necessary specialists (e.g., speech therapist), and the children's
346
Journal of School Psychology
parent(s) also involved as team members. Prior to the 1979-80 school year, each team received training in the six-step decision-analytic approach described above by means of four 2-hour workshops conducted by the author. Workshops consisted of didactic presentations, skill exercises in program planning, conflict management exercises, and role playin$ in how to involve parents and teachers in the decision-analytic process (training manual is available from author). Each team was requested by the director of special services of the district to use the approach for planning all new IEPs that were to be developed during the 10-month school year.
Outcome Measures and Measurement Procedures Four outcome measures, judged by a panel of directors of special services and state department of education officials as valid indicators, were used for the evaluation. Percentage o f team use of decision analysis. This measure reflected the extent to which the decision-analytic approach was used by both teams during the 1979-80 school year for all new IEPs to be developed. As per district policy, for each new IEP developed, both teams submitted to the director of special services a n " I E P Planning Form" that included information about the nature of the program chosen and the various program alternatives considered (copy of the form available upon request from the author). These forms were used to calculate the percentage of cases in which the approach was used by each team, given the number of new IEPs that were developed and implemented by each team. Mean number of program alternatives considered per IEP. This measure reflected the mean number of program alternatives considered by each team for each new IEP. This information was obtained from the " I E P Planning Form" for all new IEPs developed during the 1978-79 (baseline) and 1979-80 (intervention) school years. ~ ~' Percentage o f complete IEPs. This measure reflected the extent to which each new IEP ~tas complete, i.e., met seven criteria for a complete IEP, described in detail elsewhere (Maher, 1980): pupil data base, goal structure, goal indicators, program structure, evaluation procedures, description of regular education participation, date for annual review. All new IEPs developed during baseline and intervention years were rated for completeness, using the seven criteria, by two outside raters. An IEP was considered to be complete only when all seven criteria were met. Interrater agreement was calculated, relative to the overall rating for each new IEP, by means of a ratio of agreements over agreements plus disagreements multiplied times 100. Interrater agreement was 95°70 for the 1978-79 year and 98070 for the 1979-80 year. Team conflict situations. This measure assessed the extent to which a team reached an impasse, among themselves or with a parent, over the nature of an IEP. An impasse was defined as a situation where an IEP had to be referred to the director of special services for conflict resolution as determined by the director's minutes of subsequent conflict resolution meetings.
Results During 1978-79, the Elementary School Team developed 23 new IEPs, while 24 new IEPs were developed by that group during 1979-80. The Secondary School Team developed 21 new IEPs during 1979-79 and 23 new IEPs during 1979-80. Thus,
Maher
347
during both school years, each team was similar in terms of the number of new IEPs developed. Percentage of team use o f decision analysis. During 1979-80, the Elementary School Team utilized the decision-analytic procedure with all 24 new IEPs, while the Secondary School Team used the approach with all 23 new IEPs. Thus, for both teams, the team utilization rate was 100gaercent, indicating that the approach was consistently used throughout the school 'year. Mean number of program alternatives considered per IEP. During 1978-79, the Elementary School Team considered an average of 1.8 program alternatives per IEP planning situation, while during 1979-80, the average number of program alternatives considered was 3.2. For the Secondary School Team, the average number of program alternatives considered was 1.9 during 1978-79 and 3.4 during 1979-80. Percentage o f complete IEPs. During 1978-79, 14 of the 23 new IEPs developed by the Elementary School Team were complete, for a completion rate of 61°70. During 1979-80, 22 out of 24 IEPs were completed, for a rate of 92070. For the Secondary School Team, during 197g,.79, 10 of 21 new EIPs were found to be complete, for a rate of 48070, while during 1979-80, 20 of 23 IEPs were complete, for a rate of 88070. Team conflict situations. During 1978-79, the Elementary School Team was involved in situations where 6 of 23 IEPs required resolution by the director. Of the 6 situations, all requests for resolution were due to failure of the team to consider alternative placements; 4 were initiated by parents, and 2 by school principals. During 1979-80, only 1 of 24 IEPs was referred to the director. For the Secondary School Team, 5 team conflict situations occurred out of 21 IEPs during 1978-79. All situations involved conflict over placement; 3 were initiated by parents, while 2 were initiated by principals. No conflict situations occurred for 23 IEPs in 1979-80. Social validation. At the conclusion of 1979-80, individual interviews were conducted by outside interviewers (graduate students) with the school principals (n =6) to assess their perceptions of the decision-analytic approach. Responses revealed much satisfaction on the part of the administrators with the procedure, and all recommended it be continued in their schools. All core members of both teams (n = 6) were interviewed. Team members perceived the approach as being a practical one; as providing them with necessary structure for focused team discussion of the IEP; and as useful to them in planning resource rooms and drug abuse prevention programs, besides IEPs. Also interviewed was a random sample of five elementary and five secondary school parents who had been involved in the decision-analytic procedure. Eight of the ten parents indicated that the approach allowed them to appreciate more fully the complexities of planning a program for their child and allowed them a chance to understand more fully the program choices available. DISCUSSION AND CONSIDERATIONS The formative evaluation results suggest that decision analysis is a potentially useful approach for MDTs in planning special service programs and one that seems capable of being applied by MDTs for IEP planning without any apparent negative side-effects (e.g., grievances from teachers or parents about its use). Interview responses of school administrators and team members seem to suggest that the approach is practical, while parents expressed satisfaction with their involvement in
348
Journal of School Psychology
it. When compared to the district's traditional approach, decision analysis seems to have enabled both teams to consider more program alternatives for each,child, to develop more complete IEPs and to resolve conflict over program planning situations to a greater extent. There are sevgral possible advantages for MDTs in using decision analysis. First, the essential elements of planning decision can be considered separately. Thus, rather than an MDT discussing a program to be planned from a global perspective, team consensus is sought on what specifically is to be accomplished (i.e., outcomes), the available means to accomplish those ends (i.e., program alternatives), and the likelihood (probability) and value (utility) of each program choice in terms of reaching the outcomes. Second, the approach enables an MDT to consider attitudes toward risky choices, i.e., those alternatives that may not have a high probability of success. For example, the ethical and legal implications of selecting program alternatives judged to have little chance of success or that may lead to certain negative side effects can be discussed by the team, teachers, and parents, with additional program options being considered and previously selected options being reassessed in light of discussion. Third, in terms of program evaluation, the approach can help focus evaluation activities on imlaortant areas. For example, if based upon a priori probabilities of the various program alternatives generated using the approach, an MDT proceeded to implement a choice with an 80070 probability of achieving particular IEP goals, subsequent IEP evaluation could estimate the probability that goals were attained. This kind of evaluation, requiring Bayesian statistics to account for prior and posterior probabilities, would confirm the utility of a program alternative or suggest reassessment of the options (see Edwards, Guttentag, & Snapper, 1975). It should be noted here, however, that this kind of analysis was not undertaken in the exploratory investigation reported above, but should be included in future studies. Decision analysis is not a single solution to planning special service programs; several limitations and cautions should be noted. First, the procedure requires commitment, training, and coordinated effort on the part of an MDT. Experience suggests the approach to be most useful with teams that value data-based planning approaches and who have received proper training, and to be less useful with teams that do not see the approach as credible and/or who have not been trained. Second, determining outcome criteria can pose problems for team members. Thus, attention must be given by team members to deciding whether certain outcomes carry greater weight than others. If certain outcomes carry a greater weight, an explicit weighting procedure must be agreed upon (see Keeney & Raiffa, 1976). Third, assessing probabilities and utilities can be problematical, since assignment of numerical values to program outcomes may be foreign to team members. Moreover, since utilities do not sum to 100, as do probabilities, certain individuals can " l o a d " high utilities on particular program alternatives, thereby resulting in other alternatives receiving trivial weights that can eliminate their influence on a decision. Therefore, the importance of not judging certain alternatives to be 5 or 6 times more valuable than other alternatives must be understood and appreciated by team members. If this problem were to persist, and if a clearer picture of the relative values of utilities were needed, the actual utility values could be normalized whereby the utilities are summed, with each value divided by the sum and multiplied times 100 (Keeney & Raiffa, 1976).
Maher
349
Judgments about the utility of decision analysis for MDTs can only be made in a tentative way at this time. Clearly, investigations of a molar nature, comparing decision analysis to more traditional program planning approaches along a range of outcome dimensions, are needed before dismantling research strategies should be considered. Most importantly, additiona!investigationsmust begin to assess directly the ways in which decision analysis can h~lp plan programs that enhance the learning of children. Future studies might also seek to determine characteristics of public school organizations (e.g., ability of the organization to commit resources for training, values of team members) which might relate to successful use of the decision-analytic approach. REFERENCE NOTE 1. Butler, A. A conflict management program for school psychologists. Unpublished doctoral dissertation. Rutgers - - The State University, May, 1979. ~ N C E S Delbecq, A.L., Van de Van, A.H., & Gustafson, D.H. Group techniques for program planning: A guide to nominal group and delphi processes. Oakland, N.J.: Scott, Foresman and Company, 1975. Edwards, W., Guttentag, M., & Snapper, K. A decision theoretic approach to evaluation research. Handbook of evaluation research (Vol. I). Beverly Hills, Ca: Sage Publications, 1975. Kenney, R.L., & Raiffa, H. Decisions with multiple objectives: Preferences and value tradeoffs. New York: John Wiley & Sons, 1976. Maher, C.A. Training special service teams to develop IEP's. Exceptional Children, 1980, 47, 206-211. Maher, C.A., & Barbrack, C.R. Special service program evaluation: Perspectiveand principles. Journal of Special Education, 1979, 13, 413-420. Mitroff, I.I., Emshoff, J.R., & Kilmann, R.H. Assumptional analysis: A methodology for strategic problem solving. In L. Datta & R. Perloff (Eds.), Improving evaluations. Beverly Hills, Ca: Sage Publications, 1979. Yoshida, R.K., Fenton, K.S., Maxwell, J.P., & Kaufman, M.J. Group decision making in the planning team process: Myth or reality? Journal of School Psychology, 1978, 16, 237-244.
Charles A. Maher Assistant Professoff Department of School Psychology Rutgers University Busch Campus, PO Box 819 Piscataway, NJ 08854 Manuscript received: May 8, 1980 Revision received: April 21, 1981
0022-4405/81/1600-0340500.95 © 1981 The Journal of School Psychology, Inc.
D E C I S I O N A N A L Y S I S : A N A P P R O A C H FOR M U L T I D I S C I P L I N A R Y T E A M S IN P L A N N I N G S P E C I A L SERVICE P R O G R A M S CHARLES A. MAHER Rutgers University
Summary: A decision analytic approach for use by multidisciplinary teams (MDTs) in planning special service programs is described, and formative evaluation information on its application by MDTs in planning individualized education programs (IEPs) is presented. Decision analysis is applied following initial determination of program goals and objectives. In~using the approach, an MDT engages in a six-step process: (a) delineation of program alternatives, (b) determination of program outcomes, (c) assessment of probabilities, (d) assessment of utilities, (e) determination of overall program values, and (0 selection of the program to be developed. Utilization of the approach allows a complex program planning situation to be disaggregated into its essential elements, with discussion of program alternatives occurring relative to specific decision criteria. Strengths and limitations of the approach also are discussed, and future directions for research are briefly noted. The planning of special service programs by multidisciplinary teams (MDTs) is a complex decision making endeavor (Yoshida, Fcaton, Maxwell, & Kaufman, 1978). Whether the program to be planned is an individualized education program (IEP), a resource room, or an alternate school, it is unlikely that an MDT will have complete understanding of all decisions to be made or accurate knowledge of all future events likely to affect its planning decisions (Maher & Barbrack, 1979; Mitroff, Emshoff, & Kilmann, 1979). The complexity and uncertainty of special service program planning can be seen by examining the kinds of decision questions an MDT might consider when planning an IEP for a handicapped child: On what goals and objectives will the IEP focus? What alternative programs may be developed? What positive and negative effects might accrue from each alternative? Which alternative is seen as more worthwhile by professionals, parents, child? Analysis and synthesis of information for an informed program choice is not a straightforward matter for an MDT. Typically, integration of decision making information occurs in a tacit fashion, largely because the team does not possess a framework for disaggregating a global planning decision into its essential elements. Although the art of decision making should not be minimized, decision making units such as MDTs may improve their planning ability by means of decision analysis, a planning procedure derived from decision theory (see Keeney & Ralffa, 1976). In this paper, a decision analytic procedure for use by MDTs in planning special service programs is described, formative evaluation information to support the potential utility of the approach is presented, strengths and limitations of the approach are discussed, and directions for future research are noted. 340
Maher
341
THE DECISION-ANALYTIC APPROACH Decision analysis is described and illustrated below as a six-step approach: (a) delineation of program alternatives, (b) determination of program outcomes, (c) assessment of probabilities, (d) assessment of utilities, (e) determination of overall program values, and (f) selection of theiprogram to be developed. The approach is used by an MDT following identification and assessment of a program client (i.e., individual pupil, group of pupils) and the setting of goals and objectives for that client. For IEPs, decision analysis is applied subsequent to the formulation of IEP goals and objectives by the team, but prior to the placement decision. For' group programs (e.g., resource rooms), the approach is used following discussion of general goals (procedure manual is available for author). The approach has been determined by state department of education officials as being congruent with the IEP provisions of PL 94-142.
Delineation o f Program Alternatives In this step, the MDT, based upon previously established goals and objectives, delineates the program alternatives to be considered, portraying each option by means of a decision tree (see Figure 1). A decision tree provides clarity to the program planning situation: Besides depicting program alternatives, a decision tree displays the level of outcome expected to result from each alternative and the probabilities and utilities associated with each alternative. A decision tree is composed of a series of nodes and branches. Square nodes represent events that are dictated by the MDT, while round nodes are dictated by chance. In using the decision analytic approach, the sum of the probabilities assigned to each outcome for each program alternative must equal one (see Figure 1). Illustration. A fourth-grade child in a regular education classroom had been identified by the MDT as being eligible to receive special education and related services. The assessment process revealed three areas of need: reading, language arts, and behavioral self-control; and one area of strength: grade-level performance in mathematics. IEP goals and objectives were set in reading, spelling, grammar, and completion of class assignments. Using decision analysis, the MDT, in concert with the child's parents, proposed the two individualized education program alternatives seen in Figure 1: (a) special classroom program, with mainstreaming into music, art, and physical e~ucation; and (b) regular classroom program, with academic instruction in a resource room as well as behavioral self-control education program to be provided by the school counselor.
Determination of Program Outcomes In this step, expected outcomes for each program alternative are determined. Outcomes are observable, measurable criteria, such as IEP objectives, or other indicators that are developed if current goals and objectives are not considered as being amenable to an objective assessment by outside raters. In determining program outcomes, an MDT must consider program risks and benefits. In this regard, a program may improve a client's performance (benefit) or result in negative side effects (risks). For example, while a program for a group of gifted children may be
342
Journal of School Psychology
beneficial in terms of achievement of academic outcomes, the program may be seen as~placing considerable stress on the children whereby negative side effects such as cheating, psychosomatic complaints, or refusal to attend school may develop. Benefits (positive outcomes) of a program alternative can be measured in terms of prevention or (emediation of social or emotional problems, or with respect to acquisition or development of behavioral or academic skills. Risks (negative side effects) resulting from a program can be measured in terms of variables such as increased frustration, refusal to attend school, or academic failure. Illustration. The MDT further operationalized the child's IEP objectives as program outcomes by means of goal attainment scaling, a procedure which specifies levels of outcome (see Maher & Barbrack, 1979). The team decided that a "satisfactory" outcome would be attainment of 75°70 or more of all IEP objectives, at the time of IEP annual review; a "not satisfactory" outcome was considered to be less than 75070 attainment of all goals. Program benefits were considered to be achievement of satisfactory outcomes, while negative effects were seen as less than satisfactory outcomes as well as any maladaptive classroom behavior the child may exhibit.
Assessment o f Probabilities In this step, the probability that each program alternative will result in a particular program outcome is assessed by an MDT. Decision analysis requires greater attention and precision in the assessment of probabilities than usually is the case with MDTs. Often, in special service program planning situations, team members speak of positive program outcomes using terms such as frequent, satisfactory, or high. For each of these qualitative likelihood descriptions, however, a team can set a probability at 1 in 10, another at 1 in 100, or 1 in 1,000. The same kinds of probabilities can be set for negative outcome" descriptions such as uncommon, infrequent, not satisfactory, or low. To the extent possible in assessment of probabilities by an MDT, empirical evidence derived from case studies and more controlled types of program evaluations can serve as the basis for determining the probability of success of program alternatives. Experience dictates, however, that such objective data l ~ y not always be available and, when available, may not be applicable to the program client of interest. Many factors such as the age of the client, the presence or absence of particular psychoeducational strengths and needs, and previous learning experiences may need to be considered in determining probabilities of program alternatives for a particular client. Thus, in decision analysis, probability values are derived from subjective estimates of team members. This can be accomplished by averaging the estimates of individual probability values of team members, or by means of team consensus on a particular value following discussion. If team members disagree on a particular value, they can first discuss their differences and arrive at a compromise value. If disagreement persists, the matter can be referred to another party (e.g., director of special services) for conflict resolution. Procedures for obtaining team consensus on probability values and utility values (to be discussed below) can be found in Butler (Note 1), Delbecq, Van de Ven, and Gustafson (1975), and Keeney and Raiffa (1976). Illustration. The MDT assessed the likelihood of occurrence of the outcomes relating to each program alternative and obtained consensus on those likelihoods
343
Maher PROGRAM ALTERNATIVES
LEVEL OF GOAL A'rrAINMEUT (OUTCOME)
UTILITIES
SATISFACTORY(75%or more) ~ probability= .80
40
OVERALL VALUE
.80 (40) = 32
SPECIALCLASSPROGRAM NOTSATISFACTORY(less than 75%) probability .20
3o
=
I
SATISFACTORY(750 or more)
80
.20 (3o) --s o.v. *= as
.40 (80) = 32
REGULARCLASSPROGRAM(~ probability= .40
| NOT~SA?ISFACTORY : .60
(Less than
probability
75%)
.60 (so) = 36 60
O.V.* = 6g
*O.V. = ~ (Probability of outcome X utility of outcome)
Figure 1. A decision tree for a planning situation involving an IEP.
(see Figure 1). Assessment of probabilities was based not only on historical data (e.g., previous standardized and criterion referenced test scores), but also on the judgments of team members about the child which were derived from their experiences with similar kinds of children they had encountered in the past. Thus, previous learning history of the child, parental opinions, adequacy of the school district's resources, and other factors influenced team members' judgments about the likelihood of each possible outcome.
Assessment of Utilities In this step, an MDT assessess the utility of each program alternative. Assessment of utilities, however, is more complex than that of probabilities, since utilities are indices of value, and judgments of value are essentially subjective in nature. As noted above, various benefits and risks can result from different program alternatives. For a program client, benefits may include a certain level of academic achievement; risks may include potential physical or emotional discomfort, loss of self-esteem, or frustration resulting from inappropriate task demands. In any given program planning situation, several benefits and risks can be anticipated by an MDT as a consequence of a particular program alternative. Thus, it is important for team members to be able to compare the utilities of a number of benefits and risks with differing characteristics. For example, for a particular handicapped child, it may be necessary to compare the potential academic achievement (benefit) attained by the child by means of a special classroom program with the potential ridicule (risks) the
344
Journal of School Psychology
child might receive from nonhandicapped peers and the effect of such derision on the handicapped child's self-esteem. When program outcomes can be given a utility measurement in a single unit such as monetary cost, unidimensional utilities can be used, and they are readily ordered and compared. However, special service program outcomes are multidimensional. Since program outcomes such as academic achievement, behavioral self-control, and physical well-being are measured in different units, comparison and ordering of them by an MDT requires that a common multidimensional utility be devised to express all of them. For example, in planning an alternative school program for a group of socially maladjusted adolescents, if dropping out of school were given a utility value of 0 units, a particular rate of school attendance might be assigned 60 units, while a certain level of academic achievement might be given a utility value of 100 units. Though assessment of probabilities usually is the exclusive responsibility of the team, assessment of utilities often is undertaken in concert with the child and the child's parents. Since different persons assess the value of outcomes differently, it is essential when assigning utilities to agree upon rules for such assignment. For example, the rule for assignment of utilities may im/olve computing the mean of individual team members' utilities, or the rule may involve team discussion and consensus on a compromise utility. Illustration. In this example, even though the probability of satisfactory level of goal attainment for the special classroom program was .80, it had limited value to some team members due to their concern that the child would not be mainstreamed into regular academic classrooms. Moreover, although the probability of satisfactory level of goal attainment in the regular classroom was .40, the utility was seen as high by all team members since the child would be more involved in the mainstream of public school education (Figure 1). From this case example, it is apparent that the~relative worth of each program alternative is a function of both the probability of the outcome of the alternative and the utility of the outcome. Thus, in order to obtain an index of the value of each outcome, it was necessary for the team to develop a scale for assessment of the utility of each outcome. In this example, an arbitrary 100 point scale of relative units was developed by the team. On this scale, a satisfactory level of goal attainment in the special classroom program had a utility of 40 units, while a satisfactory level of goal attainment for the regular classroom program had a utility of 80 units, even though the probability for satisfactory goal attainment was higher in the special classroom program (Figure 1).
Determination o f Overall Program Values Having assigned both probabilities and utilities for each of the outcomes, the next step for a team is to calculate the overall value of each outcome for each program alternative. The value of each outcome is calculated by multiplying the probability of the outcome and the corresponding utility of that outcome. When the values of both branches are calculated, the sum of these values represents the overall expected value for that branch. Illustration. In this case, the overall value for the special class program was 38, since 40 (the utility) times .80 (the probability) equaled 32 (see last column in Figure 1), and 30 (utility) times .20 (probability) equaled 6 (see last column in Figure 1). Thus, the overall value of 32 plus 6 equaled 38 (see last column in Figure 1).
Maher
345
Selection of the Program to be Developed This step involves selection of the " b e s t " program alternative, that is, the alternative with the highest expected overall value. In using decision analysis, opinion about the program selected also can be sought from those not involved in the process (e.g., administrators), especially when it is not clear that a particular alternative is the " b e s t " one (e.g., when ~a program with a low probability has been choosen). If outside individuals or groups also choose the same alternative, the team can be more certain that their alternative is acceptable. If consensus is not obtained, a joint committee could attempt to resolve differences. Illustration. In this case example, the overall value of the regular education program was 68, signifying that the best choice was the regular classroom option. Consultation with parents and with special education and regular education teachers also verified this choice.
Planning Group Educational Programs Decision analysis also can be used by ~n MDT in planning a group program; one such example is briefly summarized here, without presentation of the actual data: Ten high school pupils had been identified by an MDT as being in need of special education services, focusing on a program goal of social adjustment. Using decision analysis, three group program alternatives were considered by the team in relation to the overall goal: (a) a self-contained special education classroom program; (b) a career-oriented program in the regular high school, emphasizing vocational skill development training; and (c) a regular high school program, with remedial instruction in needed areas. The MDT then determined specific outcomes having to do with level of attainment of social goals (e.g., improved peer relationships), assigning numerical probabilities and utilities to each outcome, for each alternative. Subsequent calculations revealed: (a) the regular classroom alternative with the vocational skill development training, although having a lower probability for goal attainment than did the special class alternative, had a much higher utility value. (b) Although the self-contained class program alternative had a high probability with respect to satisfactory goal attainment, the utility of this alternative was relatively low since team members believed that such a program would result in school truancy and lowered self-esteem. (c) The regular classroom alternative with remedial instruction was [een as being low both from probability and utility perspectives. Further calculation of the overall value of each alternative revealed the careeroriented program as the best program choice. USE OF DECISION ANALYSIS BY MDTs: A FORMATIVE EVALUATION A formative evaluation, offering initial exploratory evidence of the potential usefulness of decision analysis, is presented here: An urban public school district in New Jersey employed two full-time MDTs that had worked together for three years: (a) an Elementary School Team, responsible for planning special services in the district's four elementary schools; and (b) a Secondary School Team, with similar responsibilities in the district's middle and high school. Each team consisted of a "core team" of school psychologist, learning consultant, and social worker, with the school principal, necessary specialists (e.g., speech therapist), and the children's
346
Journal of School Psychology
parent(s) also involved as team members. Prior to the 1979-80 school year, each team received training in the six-step decision-analytic approach described above by means of four 2-hour workshops conducted by the author. Workshops consisted of didactic presentations, skill exercises in program planning, conflict management exercises, and role playin$ in how to involve parents and teachers in the decision-analytic process (training manual is available from author). Each team was requested by the director of special services of the district to use the approach for planning all new IEPs that were to be developed during the 10-month school year.
Outcome Measures and Measurement Procedures Four outcome measures, judged by a panel of directors of special services and state department of education officials as valid indicators, were used for the evaluation. Percentage o f team use of decision analysis. This measure reflected the extent to which the decision-analytic approach was used by both teams during the 1979-80 school year for all new IEPs to be developed. As per district policy, for each new IEP developed, both teams submitted to the director of special services a n " I E P Planning Form" that included information about the nature of the program chosen and the various program alternatives considered (copy of the form available upon request from the author). These forms were used to calculate the percentage of cases in which the approach was used by each team, given the number of new IEPs that were developed and implemented by each team. Mean number of program alternatives considered per IEP. This measure reflected the mean number of program alternatives considered by each team for each new IEP. This information was obtained from the " I E P Planning Form" for all new IEPs developed during the 1978-79 (baseline) and 1979-80 (intervention) school years. ~ ~' Percentage o f complete IEPs. This measure reflected the extent to which each new IEP ~tas complete, i.e., met seven criteria for a complete IEP, described in detail elsewhere (Maher, 1980): pupil data base, goal structure, goal indicators, program structure, evaluation procedures, description of regular education participation, date for annual review. All new IEPs developed during baseline and intervention years were rated for completeness, using the seven criteria, by two outside raters. An IEP was considered to be complete only when all seven criteria were met. Interrater agreement was calculated, relative to the overall rating for each new IEP, by means of a ratio of agreements over agreements plus disagreements multiplied times 100. Interrater agreement was 95°70 for the 1978-79 year and 98070 for the 1979-80 year. Team conflict situations. This measure assessed the extent to which a team reached an impasse, among themselves or with a parent, over the nature of an IEP. An impasse was defined as a situation where an IEP had to be referred to the director of special services for conflict resolution as determined by the director's minutes of subsequent conflict resolution meetings.
Results During 1978-79, the Elementary School Team developed 23 new IEPs, while 24 new IEPs were developed by that group during 1979-80. The Secondary School Team developed 21 new IEPs during 1979-79 and 23 new IEPs during 1979-80. Thus,
Maher
347
during both school years, each team was similar in terms of the number of new IEPs developed. Percentage of team use o f decision analysis. During 1979-80, the Elementary School Team utilized the decision-analytic procedure with all 24 new IEPs, while the Secondary School Team used the approach with all 23 new IEPs. Thus, for both teams, the team utilization rate was 100gaercent, indicating that the approach was consistently used throughout the school 'year. Mean number of program alternatives considered per IEP. During 1978-79, the Elementary School Team considered an average of 1.8 program alternatives per IEP planning situation, while during 1979-80, the average number of program alternatives considered was 3.2. For the Secondary School Team, the average number of program alternatives considered was 1.9 during 1978-79 and 3.4 during 1979-80. Percentage o f complete IEPs. During 1978-79, 14 of the 23 new IEPs developed by the Elementary School Team were complete, for a completion rate of 61°70. During 1979-80, 22 out of 24 IEPs were completed, for a rate of 92070. For the Secondary School Team, during 197g,.79, 10 of 21 new EIPs were found to be complete, for a rate of 48070, while during 1979-80, 20 of 23 IEPs were complete, for a rate of 88070. Team conflict situations. During 1978-79, the Elementary School Team was involved in situations where 6 of 23 IEPs required resolution by the director. Of the 6 situations, all requests for resolution were due to failure of the team to consider alternative placements; 4 were initiated by parents, and 2 by school principals. During 1979-80, only 1 of 24 IEPs was referred to the director. For the Secondary School Team, 5 team conflict situations occurred out of 21 IEPs during 1978-79. All situations involved conflict over placement; 3 were initiated by parents, while 2 were initiated by principals. No conflict situations occurred for 23 IEPs in 1979-80. Social validation. At the conclusion of 1979-80, individual interviews were conducted by outside interviewers (graduate students) with the school principals (n =6) to assess their perceptions of the decision-analytic approach. Responses revealed much satisfaction on the part of the administrators with the procedure, and all recommended it be continued in their schools. All core members of both teams (n = 6) were interviewed. Team members perceived the approach as being a practical one; as providing them with necessary structure for focused team discussion of the IEP; and as useful to them in planning resource rooms and drug abuse prevention programs, besides IEPs. Also interviewed was a random sample of five elementary and five secondary school parents who had been involved in the decision-analytic procedure. Eight of the ten parents indicated that the approach allowed them to appreciate more fully the complexities of planning a program for their child and allowed them a chance to understand more fully the program choices available. DISCUSSION AND CONSIDERATIONS The formative evaluation results suggest that decision analysis is a potentially useful approach for MDTs in planning special service programs and one that seems capable of being applied by MDTs for IEP planning without any apparent negative side-effects (e.g., grievances from teachers or parents about its use). Interview responses of school administrators and team members seem to suggest that the approach is practical, while parents expressed satisfaction with their involvement in
348
Journal of School Psychology
it. When compared to the district's traditional approach, decision analysis seems to have enabled both teams to consider more program alternatives for each,child, to develop more complete IEPs and to resolve conflict over program planning situations to a greater extent. There are sevgral possible advantages for MDTs in using decision analysis. First, the essential elements of planning decision can be considered separately. Thus, rather than an MDT discussing a program to be planned from a global perspective, team consensus is sought on what specifically is to be accomplished (i.e., outcomes), the available means to accomplish those ends (i.e., program alternatives), and the likelihood (probability) and value (utility) of each program choice in terms of reaching the outcomes. Second, the approach enables an MDT to consider attitudes toward risky choices, i.e., those alternatives that may not have a high probability of success. For example, the ethical and legal implications of selecting program alternatives judged to have little chance of success or that may lead to certain negative side effects can be discussed by the team, teachers, and parents, with additional program options being considered and previously selected options being reassessed in light of discussion. Third, in terms of program evaluation, the approach can help focus evaluation activities on imlaortant areas. For example, if based upon a priori probabilities of the various program alternatives generated using the approach, an MDT proceeded to implement a choice with an 80070 probability of achieving particular IEP goals, subsequent IEP evaluation could estimate the probability that goals were attained. This kind of evaluation, requiring Bayesian statistics to account for prior and posterior probabilities, would confirm the utility of a program alternative or suggest reassessment of the options (see Edwards, Guttentag, & Snapper, 1975). It should be noted here, however, that this kind of analysis was not undertaken in the exploratory investigation reported above, but should be included in future studies. Decision analysis is not a single solution to planning special service programs; several limitations and cautions should be noted. First, the procedure requires commitment, training, and coordinated effort on the part of an MDT. Experience suggests the approach to be most useful with teams that value data-based planning approaches and who have received proper training, and to be less useful with teams that do not see the approach as credible and/or who have not been trained. Second, determining outcome criteria can pose problems for team members. Thus, attention must be given by team members to deciding whether certain outcomes carry greater weight than others. If certain outcomes carry a greater weight, an explicit weighting procedure must be agreed upon (see Keeney & Raiffa, 1976). Third, assessing probabilities and utilities can be problematical, since assignment of numerical values to program outcomes may be foreign to team members. Moreover, since utilities do not sum to 100, as do probabilities, certain individuals can " l o a d " high utilities on particular program alternatives, thereby resulting in other alternatives receiving trivial weights that can eliminate their influence on a decision. Therefore, the importance of not judging certain alternatives to be 5 or 6 times more valuable than other alternatives must be understood and appreciated by team members. If this problem were to persist, and if a clearer picture of the relative values of utilities were needed, the actual utility values could be normalized whereby the utilities are summed, with each value divided by the sum and multiplied times 100 (Keeney & Raiffa, 1976).
Maher
349
Judgments about the utility of decision analysis for MDTs can only be made in a tentative way at this time. Clearly, investigations of a molar nature, comparing decision analysis to more traditional program planning approaches along a range of outcome dimensions, are needed before dismantling research strategies should be considered. Most importantly, additiona!investigationsmust begin to assess directly the ways in which decision analysis can h~lp plan programs that enhance the learning of children. Future studies might also seek to determine characteristics of public school organizations (e.g., ability of the organization to commit resources for training, values of team members) which might relate to successful use of the decision-analytic approach. REFERENCE NOTE 1. Butler, A. A conflict management program for school psychologists. Unpublished doctoral dissertation. Rutgers - - The State University, May, 1979. ~ N C E S Delbecq, A.L., Van de Van, A.H., & Gustafson, D.H. Group techniques for program planning: A guide to nominal group and delphi processes. Oakland, N.J.: Scott, Foresman and Company, 1975. Edwards, W., Guttentag, M., & Snapper, K. A decision theoretic approach to evaluation research. Handbook of evaluation research (Vol. I). Beverly Hills, Ca: Sage Publications, 1975. Kenney, R.L., & Raiffa, H. Decisions with multiple objectives: Preferences and value tradeoffs. New York: John Wiley & Sons, 1976. Maher, C.A. Training special service teams to develop IEP's. Exceptional Children, 1980, 47, 206-211. Maher, C.A., & Barbrack, C.R. Special service program evaluation: Perspectiveand principles. Journal of Special Education, 1979, 13, 413-420. Mitroff, I.I., Emshoff, J.R., & Kilmann, R.H. Assumptional analysis: A methodology for strategic problem solving. In L. Datta & R. Perloff (Eds.), Improving evaluations. Beverly Hills, Ca: Sage Publications, 1979. Yoshida, R.K., Fenton, K.S., Maxwell, J.P., & Kaufman, M.J. Group decision making in the planning team process: Myth or reality? Journal of School Psychology, 1978, 16, 237-244.
Charles A. Maher Assistant Professoff Department of School Psychology Rutgers University Busch Campus, PO Box 819 Piscataway, NJ 08854 Manuscript received: May 8, 1980 Revision received: April 21, 1981