- Email: [email protected]
Program evaluation in the health fields
B o o k Reviews
189
designation 'gifted.' Specifically she attempts to clarify reasons why parents who belong to the National Association of Gifted Children have designated their children as gifted and whether such perception created the social climate for more favorable development when compared with other gifted children who were not so labeled by their parents and school. Freeman mentions, but tends to gloss over, the shortcomings in her third 'control,' who were randomly selected and who no doubt contained some gifted children, thereby potentially confounding interpretation in some indeterminate way. The contribution this research makes to the literature is the extent to which Freeman's research may provide an accurate analysis of the social and psychological settings of the gifted. In this reviewer's interpretation she has succeeded in unraveling some of the mysteries surrounding development of the gifted. In particular her finding is that integration of the gifted in the mainstream does not adversely affect them except where there is lack of understanding by significant adults--surely circumstances which are deleterious to the development of any child. This well-written book deserves to be read by all those charged with formulating goals for the education and development of the gifted. Ira Gross Department of Psychology University of Rhode Island Kingston, R.I. 02881 Schulberg, Herbert C., & Baker, Frank (Eds.). Program Evaluation in the Health Fields (Vol. II). New York: Human Sciences Press, 1979. Pp. 467. This volume is a compendium of previously published articles on human service evaluation. It represents an updated version of its predecessor, published in 1969 (Vol. I). As one reflection of changing times in the field of evaluation this volume may be without equal. This reviewer once wrote that change in the field of evaluation has become so commonplace that it could safely be regarded as a constant. This volume confirms that what was then only a supposition has indeed become reality. A comparison of Volumes I and II reveals a number of important differences between evaluation in the 1960s and evaluation in the 1970s. These differences are not so much the result of advances in the fields of psychometrics, statistical analysis, management science and the like as they are of fundamental changes in the way the field of evaluation is being defined. More than anything else these changes have brought about an expanded array of criteria which the evaluator can use in determining the worth of merit of a human service program. Three separate conceptualizations or definitions of evaluation reflecting this expanded repertoire of the evaluator can be noted in the present volume. These conceptualizations, which appear and reappear across the twenty-seven articles comprising this volume, provide the reader with perhaps the most significant message of all: evaluation is an evolving discipline which has yet to define its outermost boundary. Creating this evolutionary and transitional picture of evaluation are the decision-oriented, value-oriented and systems-oriented definitions of evaluation which are found in this volume. Some writers in the field of evaluation have explicitly put forth the notion that the role of evaluation is to provide decision makers with information. This definition of evaluation is characterized by the notion that (a) the decision maker determines what is to be evaluated and may even choose the measures to be used; (b) the evaluator's role is that of a subservient advisor to decision makers; (c) the "work" of evaluation consists almost entirely of information gathering and reporting; (d) the information gathered should be relevant to the decision maker's needs; and (d) what information is important is dictated by the decisions to be made (Pace & Friedlander, 1978). While many of the contributions in this volume address evaluation from this perspective, selections by Levine and Rosenberg and by Kiresuk and Lund are particularly representative of this classification. Levine and Rosenberg, in an article describing a model of program evaluation patterned after hearings in a court of law, define the worth of a program as "the achievement, of specified goals (either as final outcomes or as intermediate steps in implementation); as cost of services; as satisfaction with service; or as a combination of one or more of these" (p. 110). Kiresuk and Lund subscribe to a similar definition when they describe their version of goal-attainment scaling, a process whereby client and service provider together
190
Journal of School Psychology
set goals and establish indices of goal attainment. From these and other articles embodying the decision-oriented stance the reader gains the notion that criteria for determining a program's worth or merit are derived directly from the program. The derivation of these criteria from the program, usually in the form of program intents, goals or objectives, represents a salient feature of the decision-oriented approach to evaluation found in this volume. A second definition of evaluation found in this volume is the value-oriented definition. A value-oriented approach to evaluation stresses the value judgments made in evaluating programs and describes the act of judging merit or worth as central to the role of the evaluator. With this definition some of the responsibility for deriving evaluative criteria shifts from decision maker to evaluator. The evaluator participates in the decision-making process and, thereby, comes into contact with the institutional and political pressures which may lead the decision maker tO opt for evaluation criteria that skirt or ignore key evaluation issues. Articles by Wildavasky and by Kiresuk and Lund (cited above) provide the most direct reference to this definition of evaluation. Wildavsky, in describing a hypothetical model of a self-evaluating organization, believes that " . . . t h e organization doesn't matter unless it meets social needs," thereby offering the evaluator a host of social, cultural and familial criteria for evaluating a program that are external to stated objectives. In a similar vein Kiresuk and Lund ironically claim in the same article in which they provide the decision-oriented statement cited earlier that the "...professional evaluator translates values into measurement," assuming perhaps that stated objectives always reflect desired values. This juxtaposition of viewpoints within a single article points to the emerging and sometimes conflicting notions of evaluation found elsewhere in this volume. These emerging and conflicting conceptualizations are sometimes unknown to the authors themselves, giving the reader a flavor of the stress and tensions lying beneath the surface of a field trying to define itself. A third conceptualization of evaluation that can be found in this volume is a systemsoriented notion of evaluation. A systems definition of program evaluation elucidates a view of programs as systems and reflects a theoretical/philosophical stance in the sciences called general systems theory. Within this perspective any particular program is viewed as a program within a larger system of programs which, in turn, is itself a subpart of some still larger system. Thus, a program can be seen both as a system unto itself, a whole which could be "broken down" into its component parts, and simultaneously as a component part of some larger entity or process. Articles by Baker and Northman and by Caro are representative of this definition of evaluation. Baker and Northman describe an "input-throughput-output" model for evaluating a federally-supprted city school mental health clinic, while Caro provides a particularly cogent case for integrating the processes of planning and evaluation. In both contributions the "larger scheme" into which the program fits is foremost in the mind of the evaluator. Through this part-whole awareness, change in the program itself is monitored throughout the larger program, system or organizational unit of which the program is a part. Thus, to understand the implications and ramifications of their programs these evaluators look beyond the immediate context in which their programs operate. These three conceptualizations of evaluation and the articles which represent them present the reader with a view both of what evaluation is and what evaluation is becoming. Together they illustrate well the expanding but yet uncharted territory of the field of evaluation. The net result is a sometimes useful but mostly heterogeneous collection of articles, resulting in a Volume more suited to libraries and agency bookshelves than to one's own professional library. This mixed recommendation also reflects the public health and medical focus of the contributions in this volume, many of which do not address issues of central interest to the school psychologist. To confirm this the reviewer rated each of the volume's twenty-seven selections as to their perceived usefulness to the school psychologist. The result was that 26%e of the contributions contained information that the reviewer felt the school psychologist "needed to know" (Chapters 3, 5, 6, 12, 19, 24, 25), another 18% contained information that it would be "nice" for the school psychologist to know (Chapters 1, 4, 7, 14, 23), and the remaining 56% contained information of questionable value to the school psychologist. REFEI~NCE Pace, C.R., & Friedlander J. Approaches to evaluation: Models and perspectives. In G. Hanson (Ed.), Evaluating program effectiveness: New directions for student services. San Francisco: Jossey-Bass, Inc., 1978.
B o o k Reviews
191
Gary D. Borich Department of Educational Psychology The University of Texas at Austin Austin, TX 78712 Goldfarb, W., Meyers, D., Florsheim, J., & Goldfarb, N. Psychotic Children Grown Up. A Prospective Follow-Up Study in Adolescence and Adulthood. New York: Human Sciences Press, 1978. Pp 78, cloth. As a psychologist working with handicapped children, I am keenly interested in what happens to autistic children as they grow up. Since the syndrome was identified by Kanner 0943) only 35 years ago, it is not surprising that there is a paucity of literature dealing with autistic adults. Yet anyone who is engaged in the development of educational programs for children must be cognizant of the fact that childhood is but a short and fleeting period in a person's total lifespan. Therefore, before we can develop realistic and effective programs, for autistic children, we need to be able to make educated guesses about the special needs that these children will be facing as adults. Because autistic chidren acquire new skills at such a slow pace,it is especially important that we devote the limited time and energy which is availa6ie to developing those skills which will be most useful to them in the long run. The authors of this monograph have undertaken a very important first step in helping us understand the long-term development of persons with autism. They have presented prospective data on 78 autistic children, comparing their "ego status" or adaptive functioning levels at three critical points in time: (a) upon admission to the Henry Ittleson Residential Treatment Center when the children were between 5 and 9 years old, (b) at time of discharge approximately four years later,and (c) at the end of follow-up, approximately eight and a half years later.The study, which was begun in 1953, covers twelve and a half years in the lifespan of this sample of children,with the age at end of follow-up ranging from 12½ to 29 years. As much as 2 0 % of the sample was between 23 and 29 years old by the end of the study. Thus, as the titleimplies, this study deals with what happens to autisticchildren as they move into adolescence and adnlthood. Considering the stateof the art at the time this research was begun, thisis an unusually well conceived and carefully designed study. Although the authors rely on such psychoanalytic terminology as ego status,they have defined this variable not only in behavioral terms but in such a way that it is equally relevant to the behavior of school-aged children and adults. This means that in evaluating children's progress over a 12-year period, they have used a single continuous standard, rather than separate standards which apply to differentage groupings. Ego status was defined by a five-pointscale,with totalimpairment in purposeful, independent functioning at one end of the continuum and age appropriate social and educational functioning at the other end of the scale. The major finding of the study is that ego statusof childrenon admission was the singlebest predictor of ego status 12 years later.Since IQ and language skillsboth correlatehighly with ego status,these findings are consistentwith those of Loiter, Barton, DeMyer, Norton, Allen, & Steel 0973), 0974), DeMyer, Eisenberg 0956), Goldfarh 0974) and Rutter 0965), all of w h o m have studied the relationshipbetween IQ and/or presence of speech and the child'slater adjustment. There are, in addition, some less expected findings. Whereas 24% of the children showed significant improvement both at time of discharge and by the end of follow-up, an equal number showed substantial progress by the end of follow-up even though they had not done so at time of discharge. Although the authors postulate a "sleeper effect" of the treatment program to explain this delayed progress, it would seem that maturation might be an equally plausible explanation for the progress. Such an explanation is consistent with findings by Wing (1972) and Mesibov (1980) that in many cases the severity of autism lessens as children reach adolescence. The most significant finding, in my view, is that although neurological status was linked to outcome at discharge, this relationship did not hold up at end of follow-up. This suggests to me that neurological impairment affects children's educational attainments but not their nonschool-related performance as adolescents and adults. Autistic girls usually have a poorer prognosis than boys. Yet these authors found that when girls were matched with boys on neurological status, their outcome was the same.
189
designation 'gifted.' Specifically she attempts to clarify reasons why parents who belong to the National Association of Gifted Children have designated their children as gifted and whether such perception created the social climate for more favorable development when compared with other gifted children who were not so labeled by their parents and school. Freeman mentions, but tends to gloss over, the shortcomings in her third 'control,' who were randomly selected and who no doubt contained some gifted children, thereby potentially confounding interpretation in some indeterminate way. The contribution this research makes to the literature is the extent to which Freeman's research may provide an accurate analysis of the social and psychological settings of the gifted. In this reviewer's interpretation she has succeeded in unraveling some of the mysteries surrounding development of the gifted. In particular her finding is that integration of the gifted in the mainstream does not adversely affect them except where there is lack of understanding by significant adults--surely circumstances which are deleterious to the development of any child. This well-written book deserves to be read by all those charged with formulating goals for the education and development of the gifted. Ira Gross Department of Psychology University of Rhode Island Kingston, R.I. 02881 Schulberg, Herbert C., & Baker, Frank (Eds.). Program Evaluation in the Health Fields (Vol. II). New York: Human Sciences Press, 1979. Pp. 467. This volume is a compendium of previously published articles on human service evaluation. It represents an updated version of its predecessor, published in 1969 (Vol. I). As one reflection of changing times in the field of evaluation this volume may be without equal. This reviewer once wrote that change in the field of evaluation has become so commonplace that it could safely be regarded as a constant. This volume confirms that what was then only a supposition has indeed become reality. A comparison of Volumes I and II reveals a number of important differences between evaluation in the 1960s and evaluation in the 1970s. These differences are not so much the result of advances in the fields of psychometrics, statistical analysis, management science and the like as they are of fundamental changes in the way the field of evaluation is being defined. More than anything else these changes have brought about an expanded array of criteria which the evaluator can use in determining the worth of merit of a human service program. Three separate conceptualizations or definitions of evaluation reflecting this expanded repertoire of the evaluator can be noted in the present volume. These conceptualizations, which appear and reappear across the twenty-seven articles comprising this volume, provide the reader with perhaps the most significant message of all: evaluation is an evolving discipline which has yet to define its outermost boundary. Creating this evolutionary and transitional picture of evaluation are the decision-oriented, value-oriented and systems-oriented definitions of evaluation which are found in this volume. Some writers in the field of evaluation have explicitly put forth the notion that the role of evaluation is to provide decision makers with information. This definition of evaluation is characterized by the notion that (a) the decision maker determines what is to be evaluated and may even choose the measures to be used; (b) the evaluator's role is that of a subservient advisor to decision makers; (c) the "work" of evaluation consists almost entirely of information gathering and reporting; (d) the information gathered should be relevant to the decision maker's needs; and (d) what information is important is dictated by the decisions to be made (Pace & Friedlander, 1978). While many of the contributions in this volume address evaluation from this perspective, selections by Levine and Rosenberg and by Kiresuk and Lund are particularly representative of this classification. Levine and Rosenberg, in an article describing a model of program evaluation patterned after hearings in a court of law, define the worth of a program as "the achievement, of specified goals (either as final outcomes or as intermediate steps in implementation); as cost of services; as satisfaction with service; or as a combination of one or more of these" (p. 110). Kiresuk and Lund subscribe to a similar definition when they describe their version of goal-attainment scaling, a process whereby client and service provider together
190
Journal of School Psychology
set goals and establish indices of goal attainment. From these and other articles embodying the decision-oriented stance the reader gains the notion that criteria for determining a program's worth or merit are derived directly from the program. The derivation of these criteria from the program, usually in the form of program intents, goals or objectives, represents a salient feature of the decision-oriented approach to evaluation found in this volume. A second definition of evaluation found in this volume is the value-oriented definition. A value-oriented approach to evaluation stresses the value judgments made in evaluating programs and describes the act of judging merit or worth as central to the role of the evaluator. With this definition some of the responsibility for deriving evaluative criteria shifts from decision maker to evaluator. The evaluator participates in the decision-making process and, thereby, comes into contact with the institutional and political pressures which may lead the decision maker tO opt for evaluation criteria that skirt or ignore key evaluation issues. Articles by Wildavasky and by Kiresuk and Lund (cited above) provide the most direct reference to this definition of evaluation. Wildavsky, in describing a hypothetical model of a self-evaluating organization, believes that " . . . t h e organization doesn't matter unless it meets social needs," thereby offering the evaluator a host of social, cultural and familial criteria for evaluating a program that are external to stated objectives. In a similar vein Kiresuk and Lund ironically claim in the same article in which they provide the decision-oriented statement cited earlier that the "...professional evaluator translates values into measurement," assuming perhaps that stated objectives always reflect desired values. This juxtaposition of viewpoints within a single article points to the emerging and sometimes conflicting notions of evaluation found elsewhere in this volume. These emerging and conflicting conceptualizations are sometimes unknown to the authors themselves, giving the reader a flavor of the stress and tensions lying beneath the surface of a field trying to define itself. A third conceptualization of evaluation that can be found in this volume is a systemsoriented notion of evaluation. A systems definition of program evaluation elucidates a view of programs as systems and reflects a theoretical/philosophical stance in the sciences called general systems theory. Within this perspective any particular program is viewed as a program within a larger system of programs which, in turn, is itself a subpart of some still larger system. Thus, a program can be seen both as a system unto itself, a whole which could be "broken down" into its component parts, and simultaneously as a component part of some larger entity or process. Articles by Baker and Northman and by Caro are representative of this definition of evaluation. Baker and Northman describe an "input-throughput-output" model for evaluating a federally-supprted city school mental health clinic, while Caro provides a particularly cogent case for integrating the processes of planning and evaluation. In both contributions the "larger scheme" into which the program fits is foremost in the mind of the evaluator. Through this part-whole awareness, change in the program itself is monitored throughout the larger program, system or organizational unit of which the program is a part. Thus, to understand the implications and ramifications of their programs these evaluators look beyond the immediate context in which their programs operate. These three conceptualizations of evaluation and the articles which represent them present the reader with a view both of what evaluation is and what evaluation is becoming. Together they illustrate well the expanding but yet uncharted territory of the field of evaluation. The net result is a sometimes useful but mostly heterogeneous collection of articles, resulting in a Volume more suited to libraries and agency bookshelves than to one's own professional library. This mixed recommendation also reflects the public health and medical focus of the contributions in this volume, many of which do not address issues of central interest to the school psychologist. To confirm this the reviewer rated each of the volume's twenty-seven selections as to their perceived usefulness to the school psychologist. The result was that 26%e of the contributions contained information that the reviewer felt the school psychologist "needed to know" (Chapters 3, 5, 6, 12, 19, 24, 25), another 18% contained information that it would be "nice" for the school psychologist to know (Chapters 1, 4, 7, 14, 23), and the remaining 56% contained information of questionable value to the school psychologist. REFEI~NCE Pace, C.R., & Friedlander J. Approaches to evaluation: Models and perspectives. In G. Hanson (Ed.), Evaluating program effectiveness: New directions for student services. San Francisco: Jossey-Bass, Inc., 1978.
B o o k Reviews
191
Gary D. Borich Department of Educational Psychology The University of Texas at Austin Austin, TX 78712 Goldfarb, W., Meyers, D., Florsheim, J., & Goldfarb, N. Psychotic Children Grown Up. A Prospective Follow-Up Study in Adolescence and Adulthood. New York: Human Sciences Press, 1978. Pp 78, cloth. As a psychologist working with handicapped children, I am keenly interested in what happens to autistic children as they grow up. Since the syndrome was identified by Kanner 0943) only 35 years ago, it is not surprising that there is a paucity of literature dealing with autistic adults. Yet anyone who is engaged in the development of educational programs for children must be cognizant of the fact that childhood is but a short and fleeting period in a person's total lifespan. Therefore, before we can develop realistic and effective programs, for autistic children, we need to be able to make educated guesses about the special needs that these children will be facing as adults. Because autistic chidren acquire new skills at such a slow pace,it is especially important that we devote the limited time and energy which is availa6ie to developing those skills which will be most useful to them in the long run. The authors of this monograph have undertaken a very important first step in helping us understand the long-term development of persons with autism. They have presented prospective data on 78 autistic children, comparing their "ego status" or adaptive functioning levels at three critical points in time: (a) upon admission to the Henry Ittleson Residential Treatment Center when the children were between 5 and 9 years old, (b) at time of discharge approximately four years later,and (c) at the end of follow-up, approximately eight and a half years later.The study, which was begun in 1953, covers twelve and a half years in the lifespan of this sample of children,with the age at end of follow-up ranging from 12½ to 29 years. As much as 2 0 % of the sample was between 23 and 29 years old by the end of the study. Thus, as the titleimplies, this study deals with what happens to autisticchildren as they move into adolescence and adnlthood. Considering the stateof the art at the time this research was begun, thisis an unusually well conceived and carefully designed study. Although the authors rely on such psychoanalytic terminology as ego status,they have defined this variable not only in behavioral terms but in such a way that it is equally relevant to the behavior of school-aged children and adults. This means that in evaluating children's progress over a 12-year period, they have used a single continuous standard, rather than separate standards which apply to differentage groupings. Ego status was defined by a five-pointscale,with totalimpairment in purposeful, independent functioning at one end of the continuum and age appropriate social and educational functioning at the other end of the scale. The major finding of the study is that ego statusof childrenon admission was the singlebest predictor of ego status 12 years later.Since IQ and language skillsboth correlatehighly with ego status,these findings are consistentwith those of Loiter, Barton, DeMyer, Norton, Allen, & Steel 0973), 0974), DeMyer, Eisenberg 0956), Goldfarh 0974) and Rutter 0965), all of w h o m have studied the relationshipbetween IQ and/or presence of speech and the child'slater adjustment. There are, in addition, some less expected findings. Whereas 24% of the children showed significant improvement both at time of discharge and by the end of follow-up, an equal number showed substantial progress by the end of follow-up even though they had not done so at time of discharge. Although the authors postulate a "sleeper effect" of the treatment program to explain this delayed progress, it would seem that maturation might be an equally plausible explanation for the progress. Such an explanation is consistent with findings by Wing (1972) and Mesibov (1980) that in many cases the severity of autism lessens as children reach adolescence. The most significant finding, in my view, is that although neurological status was linked to outcome at discharge, this relationship did not hold up at end of follow-up. This suggests to me that neurological impairment affects children's educational attainments but not their nonschool-related performance as adolescents and adults. Autistic girls usually have a poorer prognosis than boys. Yet these authors found that when girls were matched with boys on neurological status, their outcome was the same.