- Email: [email protected]
Methods of psychoeducational program evaluation in mental health settings
Patient Education and Counseling, Elsevier Scientific
Publishers
205
19 (1992) 205-218 Ireland Ltd.
Methods of Psychoeducational Health Settings
Program Evaluation in Mental
Joseph Walsh North Community Counseling Centers, 4897 Karl Road, Columbus, OH 43229 (USA) (Received (Accepted
March 4th, 1991) Janaury 8th, 1992)
Abstract Psychoeducational programs for families of the mentally ill became widespread during the 1980s as a means of providing a forum for the relevant education and mutual support of participants. While these programs are thought to be extremely useful as interventions, very little emphasis has been placed on evaluation as a means of demonstrating their effectiveness in achieving goals. There is a possibility, then, that psychoeducation will continue to flourish with little direct evidence ofpositive outcomes for its ,family participants. This article consists of a literature review of existing methods of psychoeducational program evaluation, both quantitative and qualitative, all of which may be applicable in certain circumstances. The process by which an evaluation instrument was developed for a program withfamilies of the mentally ill is then presented in some detail.
Keywords: Psychoeducation; strumentation.
Evaluation;
In-
Introduction Psychoeducational programs can be defined as time-limited, closed groups conducted by mental health professionals for the purposes of educating and providing support to a lay membership. Among other target populations, these groups became widespread during the 1980s in service provision to families of 0 1992 Elsevier 0738-3991/92/%05.00 Printed and Published in Ireland
Scientific
Publishers
Ireland
the mentally ill [ 11. Such groups, while similar in overall purpose, may take on a variety of formats based on size, duration, session length, specific content, range of membership (such as nuclear versus extended families, siblings, neighbors) and structure, among other variables [2]. Therefore, making comparisons between groups with regard to relative success or utility is a precarious undertaking. What is functional in one setting may not be generalizable to another. There is also no reason to assume that all psychoeducational programs should focus on similar outcomes. Still, because programs for families of the mentally ill are believed to be useful services, it is worth reviewing the types of outcomes which nurses and other professionals consider to be important, and how group leaders determine whether they are being met. The purpose of this paper is to review existing methods of psychoeducational program evaluation, the rationale for those methods, and finally to present a specific evaluation tool developed by this author. Much emphasis will be placed on the rationale for evaluation, which should always be based on the needs and interests of the group leader rather than a less focused desire to be scientific. The process of instrument development is detailed to provide the reader with a general strategy which may be applied to other program settings. A major theme of this paper is that program evaluation can be a creative process, Ltd.
206
and leaders should be reluctant to adopt instruments used in other psychoeducational programs without at least considering differences in group and organizational settings. Types of knowledge generation
Program evaluations are typically quantitative in nature. In psychoeducational programs, for example, positive outcomes may be measured in units including change scores on a pretest and posttest of knowledge about the effects of psychotropic medications, or the number of contacts which participants initiate outside the group setting with other families of mentally ill persons. However, the science of research has been undergoing a transformation during the past 20 years in recognition of the idea that qualitative methods of knowledge generation are as valid in certain circumstances as those focused on hard data [3]. The program evaluator need not feel compelled to rely only on statistical methods, as these may produce evidence of change among participants which nonetheless misses the essence of that change. Likewise, changes may take place which cannot be captured by statistical methods at all. Particularly in psychoeducational programs, where the numbers of participants are small and the group process generated is as important as particular outcomes, qualitative or case study methods can be quite useful to the professional. There are a number of emerging scientific paradigms, or ways of generating knowledge, but for purposes of this discussion only two will be outlined. The positivist paradigm, or traditional scientific method, assumes that external reality has precision as a system of distinct, observable, interacting variables which have quantity and operate in regular patterns [4]. The goal of science, which is culture and value-free, is to discover those patterns and thus explain phenomena. Evaluation in this sense is a controlled, systematic process of working with variables statistically.
Critics of this method argue that the scientific method does not translate well to the realm of human science. Any breakdown of social phenomena into discrete variables is said to be reductionistic, reflecting the biases of the researcher as much as external reality. Therefore, an interpretive paradigm has gained prominence in the social sciences which assumes that human uniqueness is found in the symbols groups use to define their realities [5]. Inductive inquiry is pursued in natural settings toward the goal of allowing subjects to develop their own outcome themes. From this perspective, the program evaluator attempts to minimize his or her own expectations of group process and outcome and allows the group to set its own priorities. Because interpretive processes will be unique to each situation, outcome themes will be different for each set of individuals participating in a psychoeducational program, even when it is conducted by the same mental health professional in the same agency. The point to be emphasized here is that the psychoeducational program evaluator should not become preoccupied with technique prior to having had group experience, and should not feel compelled to secure evidence of positive outcome within a particular methodological paradigm. Statistical methods facilit: ate the identification of trends and major categories of program impact, but qualitative methods are illuminating with the relative richness of detail they provide. Both methods have been used productively in programs for families of the mentally ill, and it is to examples of these that this narrative now turns. Examples of program evaluations
While psychoeducational programs are varied in format, it is possible to examine and compare how a number of well-known groups have been evaluated. An early group program at Thresholds, a psychosocial rehabilitation center in Chicago, compared families participating in a weekly, 12-session series of
207
meetings with a nonequivalent control group of nonparticipant families [6]. Outcome measures for the group, conducted by several mental health professionals and including the client family member, consisted of parents’ reports of significant gains in attitude restructuring toward their impaired children, and the rate of members’ children who left home during the course of the program. A short questionnaire devised by the group leaders specifically for this pilot program was used to secure parents’ feedback relative to the outcomes. The leaders admitted that their evaluation design was far from conclusive, and that they were looking somewhat nonspecifically for encouraging results utilizing subjective criteria. Another program implemented a family skills-training component along with educational approaches toward the goal of improving family stress management [7]. Single families attended a series of six weekly sessions with mental health agency professionals following the client relative’s hospital discharge. Outcome was measured in terms of rehospitalization rates during a 6-month follow-up period, using a nonequivalent control group of agency clients whose families did not participate in the program. The data were retrieved from client records and family member reports. While 104 families were eventually included in the program evaluation, it must be noted that, as a part of the study, clients in the treatment group were maintained on higher doses of medication. One purely supportive psychoeducational model sought to enhance the ability of families to cope with the stress of caring for a mentally ill member without overtly attempting to change behavior [8]. A single clinician met with a family in their home, only as often as the family requested. The professional worked to aftirm that the family was not responsible for the client member’s mental disorder, that professionals have probably not been sufficiently responsive to families, and that the clinician wanted to offer whatever support
might be helpful. Using a short satisfaction survey given as the visits ended, the programmers sought to investigate whether families were comfortable with the model and if it contributed to a reduction in subsequent hospital days for the disabled family member. The application of experimental methods to program evaluation is generally impractical for agency-based professionals, but these have been utilized by several practitioners with university affiliations. One trio of researchers randomly assigned clients with schizophrenia to four treatment groups, each of which contained some combination of family intervention, social skills training, behavior therapy, and chemotherapy 191. The psychoeducational component of their comprehensive intervention consisted of a day-long survival skills workshop in which groups of families were provided with information and an overview of the social skills training intervention for their client relatives. This 8-h session, in which a number of professionals participated, was followed by single-family sessions every 2-3 weeks for 6 months to 1 year in which information from the workshop was monitored and applied. The program was evaluated after 1 year by comparing rehospitalization rates among the four groups. Unfortunately, the effects of the psychoeducational component of the intervention could not be isolated from the other features. A more specific psychoeducational program was evaluated with an experimental design and both qualitative and quantitative methods by another group of researchers [lo]. They conducted a two-session, in-home educational program for all members of single families and included the member with schizophrenia. The first session focused on the nature of schizophrenia, and the second concentrated on issues of medication as the cornerstone of the community management of schizophrenia. Three professionals participated in the presentation; session length was not reported. The goal of this intervention was simply to educate families, based on the
208
rationale that knowledgeable families will be attuned to the warning signs and ongoing risks of relapse with the disorder of schizophrenia. The authors randomly assigned 36 families into single experimental and control groups of equal size. Control group clients were seen for individual treatment. The evaluation instrument was a two-part questionnaire designed to measure knowledge about schizophrenia. The first part consisted of six open-ended questions about diagnosis, causes of the illness, medications, and therapeutic treatment factors. An independent rater coded responses into categories based on their frequency. The second part included fourteen multiple-choice questions testing knowledge of schizophrenic symptoms, medication and other treatments, side-effects, and the effects of stress and life events on the client member. Mean scores of the responses on this part of the questionnaire were tabulated for group comparisons. Participants were tested with the same instrument immediately after the family sessions, and then 3 and 9 months later. Average scores were compared between the experimental and control groups, but their sizes were too small to permit the use of inferential statistics. The program evaluation methods outlined above utilized a variety of outcome measures and designs, but almost all were applied statistically. Many of them concentrated on client-member measures of symptomatology or rehospitalization, rather than outcomes related to the family caregivers themselves. As will be implied below, program outcomes focused on the caregivers can be more suited to the inclusion of qualitative evaluation approaches than those concerning the client relative. These inductive approaches can be quite valid when carefully applied. Two similar education and support groups for families of the mentally ill have been evaluated qualitatively. The first was a weekly, ten-session program, including approx-
imately eight families in each series, whose clients were treated at an out-patient aftercare clinic [ 111. The purposes of the program were to provide families with information about mental illness, enable them to take more time for themselves, and teach them management techniques for improving client behaviors at home. The evaluation consisted of a built-in program feature in which parents were asked each week by the professional staff to make suggestions about how future meetings or series could be made more beneficial. One of the three ongoing leaders served as a designated recorder and copied down all feedback for later discussion at the regular betweengroup staff meetings. The reasons for disapproval of any family suggestions were discussed at the psychoeducational meetings. The assumptions of these evaluators were that the of member preferences accommodation would result in maximally beneficial groups, and that members would openly provide such feedback. In fact, the program did evolve significantly through this process. The coleaders also compiled family case narratives which were content-analyzed to identify emergent themes in the program for each set of participants. The present author conducts a similar type of group program (nine weekly sessions of 90 min each, including eight to twelve participants) in which the goals of education and the production of greater family support are pursued [ 121. The initial evaluation process involved a pretest, administered by phone by the group leader to determine how much families know about their client relative’s diand treatment inagnosis, medications, volvements. The short survey questionnaire also asked families to describe their major concerns about the client member and what they hoped to gain from the group experience. A posttest survey, given at the final session, asked participants to describe in open-ended fashion anything new they had learned relative to the three areas noted above, any changes they had initiated within their
209
families or social milieus to feel less isolated and more comfortable, and any suggestions for program improvement. This rather straightforward approach was intended to provide the leader with feedback about program content and the process of sharing among members for use in the planning of future series. One shortcoming of this method was that it secured no feedback from families who dropped out of the program without explanation or notice. Some programs, of course, are not evaluated at all with regard to specific outcomes, and while this is generally an undesirable practice, it does not suggest in every case a lack of competence or ability to determine effects. Probably no one in the mental health field has done more to promote the value of the psychoeducational format than Hatfield [13], yet nowhere in her volume on program implementation theory and practice does she discuss criteria for evaluation. Still, her approach is conceptually based on a solid review of the literature, and she devotes more attention to educational theory than any other writer in this area. In her view, psychoeducation is a process that will continue for families long after their participation in her program ends. Hatfield’s emphasis on a long-term process which will utilize input from many sources, professional and otherwise, renders a limited-scope evaluation of little relevance for her. An example of instrument development
The present author’s Family Education and Support Group, noted earlier, is a timelimited program for the adult family members of a mental health agency’s severely mentally disabled clients. The nine sessions, conducted by a social worker, consist of four structured educational presentations alternating with five group discussion and problem solving meetings. After 7 years of continuous operation, with the program offered three times yearly, a decision was made to revise the
evaluation procedure by adding a quantitative component in order to compile some statistical information longitudinally. The process by which the evaluation instrument was balanced with quantitative and qualitative components is detailed here to provide the reader with one example of how the activity can unfold. The question investigated in the current evaluation project is: Do families participating in the Family Education and Support Group have more involvement with supportive individuals and groups outside the nuclear family, and less involvement with their client relative’s activities of daily living, both immediately after and 1 year after completing the program than they did prior to beginning the program? Increased involvement with social support resources, including the client’s mental health treatment providers, indicates an increase in family coping skills and participation in the client’s rehabilitation program. Less involvement with the client relative suggests reduced dependence of the client on the family. An effort is made to determine the effects of the family group, and to investigate whether the acquisition of benefits at the end of each series is maintained after 1 year. There is no intention to generalize results to any other types of psychoeducational programs. Specification of variables
The two dependent variables focused on in this program are (a) the level of independent living skill exhibited by the client relative, and (b) the level of social support secured by the family with the goal of enhancing their coping and management skills with the client relative. Both of these variables derive from the stated goals of the program. Independent living skills. Perhaps the most significant research of the past decade in the area of family interactions as influenced by chronic mental illness has centered on the effects of expressed emotion. Findings suggest
210
that the more face-to-face contact a family and client maintains, the more likely negative emotions will be expressed, and higher stress levels will be experienced by the client relative. Subsequently, there is a higher likelihood of relapse into psychosis and rehospitalization. Thus, if the client’s independent living skills increase, he or she will function less dependently on the family, and expressed emotion will be reduced. In the education component of the program, families are taught methods for enhancing the client’s development of these skills through behavioral management. A constitutive definition of independent living skills for application to the evaluation process was derived by the author from a review of the literature [9,10,13]. It was defined as the ability of a chronically mentally ill client to manage a budget, maintain physical health, keep his or her surroundings clean, acquire basic material resources, maintain supportive relationships with family and friends, engage in regular community activity, and participate regularly in a mental health treatment program. These skills can be possessed in degrees, and do not necessarily imply that the client is living on his or her own. This concept was eventually operationalized in the quantitative portion of the independent living skills inventory as a change score derived from family member perceptions of the client’s skill levels in the seven areas described above, at the end of the program and at a l-year follow-up. Social supports. Another major issue with families of the chronic mentally ill is their tendency toward social isolation and the deleterious effects this has on their ability to cope with stress [ 151. By broadening their range of supports, family members can more effectively pursue their own goals and maintain the energy and motivation to work consistently toward client member’s the rehabilitation. As well as being sources of guidance, social supports provide recreation
for the relief of tension and serve as diversions from focusing too much energy and attention on the client. Based on a second literature review [9,10,16,17], the constitutive definition of social supports in this project was all resources outside the nuclear family, including extended family, friends, co-workers, peer groups, service organizations, and mental health professionals, which provide the family the opportunity to speak freely about the stresses of living with or near a chronically mentally ill relative, and which promote each family member’s relaxation, coping skills, and physical health. Like the first variable, social supports are defined quantitatively as a change score on the social support utilization section of the instrument. Instrumentation The Family Education and Support Group program evaluation instrument, titled as a Member Assessment, is included as an appendix to this paper. It consists primarily of (a) two Likert-type scales, each of which measures characteristics of one of the variables discussed above, and (b) an equal number of open-ended questions intended to acquire more detailed, individualized responses to the scaled items. Three demographic items are included at the bottom of the form, and are formatted in a style conducive to accurate response [ 181. Likert scaling was chosen for the quantitative portion of the evaluation because of its effectiveness in measuring attitudes and affects [ 191. Family members are asked to report their perceptions of the client’s independent living skill level, which tap into feeling responses, and on their own social patterns, which are self-perceptions. Likert scaling was also chosen because Summated scales permit longitudinal comparisons. The instrument is administered to the same individuals three times, and change scores are easily computed as operational measures of each variable.
211
With this measurement format, each respondent receives two scores at each administration. The independent living skills variable, with 14 items and a O-3 range in scoring for each item (negative to positive), contains an overall scoring range of O-42. The scale measuring the variable of social support, with 10 items, has a range of O-30. The two posttest scores can be compared with one another to evaluate the group program’s impact on those variables. The narrative feedback included with each scale item can be content analyzed for additional feedback about program impact on participants. The evaluator organizes respondent comments into categories, which are not determined in advance but are based on emerging themes, and constructs a summary profile of each group. The response formats are the same for both scales. A four-point very rarely/infrequently scaling method was chosen as it permits an item wording format which reduces the possibility of socially desirable responses. That is, while individuals may be reluctant to admit that they have no access to a desirable coping resource (as in agree/disagree scaling), they may admit to utilizing the same resource infrequently. The directions for completing these sections of the instrument include a description of the variable under consideration to orient respondents to its purpose. Group members do not describe themselves in the first scale, so there is little danger of the variable notation prompting certain socially desirable responses. However, identifying the second variable, social supports, by name might elicit such a response set and thus is not included. Scale items are represented as ways of coping with stress, which seems to be a more neutral terminology. Content validity for the scales was established with the assistance of a panel of experts. Initially, 21 items were written to operationalize the independent living skills variable, and 16 were written for the social supports variable. The constitutive definition
of the former variable contained seven distinct components. Three items were written as measures of each component, all of them original and based on discussions in the literature as well as the author’s own clinical experience. The latter concept did not lend itself quite as systematically to item development. However, the 16 items were also original and based on factors included in the constitutive definition of social support. These items were felt to reflect major themes found in the literature review. Next, a document was composed for distribution to a seven-member panel of experts. The panel, consisting of mental health professionals from area agencies with at least 5 years of experience with chronically mentally ill clients and families, was asked to review the variable definitions and measurement items. The items were to be rated as 1, 2, or 3 (low to high) depending on the expert’s opinion about the relevance of each statement’s placement into its conceptual category. Panel members were also asked to comment on the wording of items. The intent was to include the two highest rated items for each of the seven components of the independent living skills variable in the final instrument. However, no statement would be used with a mean rating of less than 2.0. For the social supports variable, the plan was to include all items in the final instrument with a mean rating of 2.0 or above. The results of the content validity exercise were illuminating. For six of the seven components of the first variable, at least two items received ratings of 2.0 or higher and were thus included in the final instrument. Only one statement reflecting the remaining component received a 2.0 rating. A written suggestion by one expert that the second-highest rated statement be reworded was followed, and that item was included in the program evaluation instrument. Its rating had been only one point shy of a 2.0 rating in its original form. The social support items were apparently not as well developed as the former statements. Only 8 of
212
16 received the necessary rating score. Two experts suggested additional items to be included as measures of this variable, and this was done, bringing the total to 10. Clearly, the use of a panel of experts resulted in a significant editing of the instrument. As stated earlier, the narrative comments of group members about each of the scale items are categorized for theme identification. In this process the leader lists each participant’s feedback on a separate sheet by topic in order to construct a group profile. This author develops topics for presentation to the group based on the pretest responses. The narratives also provide more detail about any changes taking place within families over time. Family members tend to note their ongoing problems on the first posttest which helps the leader understand what issues may merit more attention in subsequent groups. One of the demographic items adds to the scope of data collected with the instrument. The item concerning place of residence is a rank-order measure, from most dependent (family) to least dependent type. If at the time of first or second posttest the client relative has moved into another level of housing, the independent living skills variable is reviewed to see if its score changes in the same direction. This demographic indicator is not used at the present time as a measure in the evaluation process, but if it seems to correlate significantly with the first variable it may be used as such in the future. Face validity of the form was examined by distributing draft copies to live former members of the Family Education and Support Group. These persons were asked in writing to comment on the clarity of the survey, the ease of its completion, and the relevance of its content. Several changes in the format and order of items resulted from this exercise. Data collection Program evaluation is to be collected at three stages: during the first meeting of the
Family Education and Support Group (the pretest), at the ninth and final group meeting (posttest), and 1 year after the final meeting, by mail (second posttest). The same instrument is used at all three data collection stages. Change scores and narrative themes are calculated for each respondent between the pretest and first posttest to determine shortterm program impact, and between the first and second posttests to evaluate long-term benefits of the program with regard to the variables under consideration. No problems have been encountered in collecting data during the first two stages. New group members are asked to complete the pretest after their introduction to the program at the first session. Individuals who for some reason miss the initial meeting are asked to complete the form during the short break during session number two, the last meeting to which new members can be admitted. Members complete the posttest at the ninth and final session. Those who miss the final session are phoned by the group leader and asked to till out the evaluation at home, and copies are sent with a stamped return envelope. A time period of 2 weeks is set for the return of mailed forms to minimize the potential effects of history on the respondents. Data collection 1 year later, which has not yet occurred, will be more complicated. During the anniversary month of each group’s termination, the leader will contact former members by phone. He will remind each family of the l-year follow-up that was discussed during the program, and announce that a copy of the member assessment form will be mailed within 2 weeks. The purpose of the follow up will be presented simply - that the agency is interested in learning about the status of each family’s situation relative to the emotionally disabled family member so as to update and improve the program’s content. Former group members have traditionally enjoyed keeping in touch with the agency following their program involvement, so no major problems are anticipated in securing
213
cooperation. One problem may involve former members who have moved. The leader maintains names and addresses on file from past groups, but there may be a significant subject mortality rate at the l-year follow-up. The Member Assessment form will be sent to former members with an accompanying cover letter and a stamped, addressed envelope. Respondents will be given 2 weeks to return the material. If it is not submitted, the leader will again call the individual and ask him or her again to complete and return the form. If still not received within 2 weeks, a second packet will be sent. As a final reminder, a post card will be sent to any nonrespondents another week later. Evaluation data collected from several groups, and covering a period of years, will hopefully become useful in monitoring program usefulness to family members. Summary
Program evaluation is within the capability of every mental health professional. Most practitioners attempt to monitor the quality of the services they provide; if not by formal measurement then through the use of clinical supervision, case consultation, and observations of client behaviors. However, as has been demonstrated in the foregoing examples, a number of adaptable techniques have been developed for psychoeducational program evaluation which can provide objective feedback on outcome attainment. Some of these involve stricter research designs than many practitioners have the inclination or authority to implement. There is, in fact, little incentive in most agencies for clinicians to engage in elaborate evaluation projects. Fortunately there are other instruments and outcome indicators which can be readily incorporated into a program structure. Ant evaluation format should be consistent with the purpose of the activity and the priority program outcomes. Evaluation designed to suit the directives of administrative bodies are
common but can be irrelevant to the concerns of group members, unless the program leader is independently motivated to examine clinical issues as well. All of the evaluations described in this paper have been focused on various outcomes relevant to the program participants. This writer is biased towards those formats which incorporate the qualitative comments of participants in addition to such statistics as client member rehospitalization, as each serve useful but somewhat different functions for the evaluator. A mix of outcome measures can be included in any evaluation design, and formats different from those outlined in this paper are likely to emerge in newer programs. Just as psychoeducation has assumed diverse forms for various target populations, there is room for innovation in the methods of its evaluation. References 1 2 3
4
5 6
Hatfield AB, Lefley HP eds: Families of the Mentally III: Coping and Adaptation. New York: Guilford, 1987. McFarlane WR ed: Family Therapy in Schizophrenia. New York: Guilford, 1983. Popkewitz T : Paradigm and Ideology in Educational Research: The Social Functions of the Intellectual. New York: The Falmer Press, 1984. Ary D, Jacobs LC, Razavieh A: Introduction to Research in Education, 3rd edition. New York: Holt, Rinehart, and Winston 1985. Lincoln YS, Guba EG: Naturalistic Inquiry. Newbury Park, CA: Sage, 1985. Dincin J et al: Restructuring parental attitudes: working with parents of the adult mentally ill. Schizophrenia Bulletin 1978; 4: 597-608. Goldstein MJ, Kopeikin HS: Short- and long-term effects of combining drugs and family therapy. In: Goldstein MJ ed. New Developments in Intervention with Families of Schizophrenics. San Francisco: Jossey-Bass, 1981. Byalin K et al: Family intervention with treatmentrefractory schizophrenics. 20th International Congress of Applied Psychology, Edinburgh, 1982. Anderson CM et al: Schizophrenia and the Family. New York: Guildford, 1986. Falloon IR et al: Family Care of Schizophrenia. New York: Guilford, 1984. McLean CM et al: Group treatment for parents of the adult mentally ill. Hosp Commun Psychiatry 1982; 33: X4-569. Walsh J: Psychoeducational program evaluation: one practical method. J Psychosocial Nurs Mental Health Services 1987; 25: 25-31.
214
13 14
15 16 17 18
Hatfield AB: Family Education in Mental Illness. New York: Guilford, 1990. Koenigsburg A, Handley R: Expressed emotion: from predictive index to clinical construct. Am J Psychiatry
19
Mueller DJ: Measuring Teacher’s College, 1986.
Social
1986; 143: 1361-1373. Vine P: Families in Pain. New York: Pantheon, 1982. Kantor JS ed: Clinical Issues in Treating the Chronic Mentally III. San Francisco: Jossey-Bass, 1985.
Correspondence
Families of the Mentally III Collective: Families Helping Families. New York: W.W. Norton, 1986. Sudman S, Bradburn NM: Asking Questions. San Francisco: Jossey-Bass, 1982.
J. Walsh North Community Counseling Centers 4897 Karl Road Columbus, OH 43229, USA
Attitudes.
New
York:
to:
Appendix: The family education and support group member assessment Name:
Date:
In this first series of statements, please assess your emotionally disabled family member’s independent living skills. Please remember that these can be possessed in degrees and do not necessarily indicate that your relative is living on his or her own at the present time. As you consider each item, use the past month as a time frame in deciding upon your response. Underline the response which most closely indicates your assessment of each statement, as demonstrated by the example. Example:
My family member is a friendly VERY RARELY OCCASIONALLY person. FREQUENTLY RARELY This respondent feels that the family member is sometimes friendly, not as friendly as he or she might be. Now, please proceed to complete the scale below, and elaborate on each of your responses in the space beneath each item. My emotionally disabled family member: 1. Uses money to pay his/her personal expenses in a reasonable way. Explain:
2. Takes care of housekeeping without being reminded. Explain:
3. Maintains Explain:
an adequately
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
tasks
balanced
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
diet.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
215
4. Arranges promptly Explain:.
for transportation when needed.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
5. Spends time with friends outside the home. Explain:
6. Engages in recreational outside the home. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
activity
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
7. Makes an effort to secure income from a source outside the family. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
8. Keeps his/her
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
clothing
clean.
Explain:
9. Maintains reasonable expectations of the family for companionship. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
10. Takes the initiative or maintain a job. Explain:
to achieve
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
as prescribed.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
11. Takes medication Explain:
216
12. Has a physician whom he/she visits when physically ill. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
13. Purchases his/her own clothing and household supplies. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
14. Meets at least monthly with a mental health counselor or case worker. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Next, rather than thinking about your family member, please consider your own situation. Listed below is a series of statements reflecting various types of personal relationships and activities. These refer to different ways in which persons may cope with the stresses associated with having a disabled family member. There are no right or wrong responses to these items, since all people cope with stress differently. Next to the statements below, please underline the response which best describes your frequency of participation. Again, elaborate on your responses in the space below each item. Example:
I exercise to work off tension.
This respondent
hardly
OCCASIONALLY RARELY
if at all. Please go on now to complete
the scale items below.
hobbies
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
in at least one social group.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
1. I engage in personal outside the home. Explain:
2. I participate community Explain:
ever gets exercise,
VERY RARELY FREQUENTLY
3. I have contact with my family member’s mental health service provider. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
4. I spend time with friends for purposes unrelated to my family problems. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
5. I can devote uninterrupted time to my job or my own routine daily activities (if unemployed outside the home). Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
6. I visit a mental health professional for help with my own problems. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
I confide in the following people about my family problems: 7. Neighbors.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Explain:
8. Extended family members (aunts, cousins, grandparents, etc). Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
9. My physician.
VERY RARELY OCCASIONALLY FREOUENTLY RARELY
Explain:
10. My pastor.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Explain:
The emotionally disabled member in my family is my: SON DAUGHTER MOTHER FATHER
BROTHER SISTER SPOUSE OTHER (please specify:)
218
He/she lives in: MY HOUSE OR APARTMENT A GROUP HOME FOR EMOTIONALLY DISABLED PERSONS AN AGENCY-MANAGED OR SUPERVISED APARTMENT HIS OR HER OWN HOUSE OR APARTMENT OTHER (please specify:) years old and has been in treatment at this or other mental health agencies or He/she is now ~ years. psychiatric hospitals for approximately _ Other comments?
THANK YOU VERY MUCH FOR YOUR COOPERATION
Publishers
205
19 (1992) 205-218 Ireland Ltd.
Methods of Psychoeducational Health Settings
Program Evaluation in Mental
Joseph Walsh North Community Counseling Centers, 4897 Karl Road, Columbus, OH 43229 (USA) (Received (Accepted
March 4th, 1991) Janaury 8th, 1992)
Abstract Psychoeducational programs for families of the mentally ill became widespread during the 1980s as a means of providing a forum for the relevant education and mutual support of participants. While these programs are thought to be extremely useful as interventions, very little emphasis has been placed on evaluation as a means of demonstrating their effectiveness in achieving goals. There is a possibility, then, that psychoeducation will continue to flourish with little direct evidence ofpositive outcomes for its ,family participants. This article consists of a literature review of existing methods of psychoeducational program evaluation, both quantitative and qualitative, all of which may be applicable in certain circumstances. The process by which an evaluation instrument was developed for a program withfamilies of the mentally ill is then presented in some detail.
Keywords: Psychoeducation; strumentation.
Evaluation;
In-
Introduction Psychoeducational programs can be defined as time-limited, closed groups conducted by mental health professionals for the purposes of educating and providing support to a lay membership. Among other target populations, these groups became widespread during the 1980s in service provision to families of 0 1992 Elsevier 0738-3991/92/%05.00 Printed and Published in Ireland
Scientific
Publishers
Ireland
the mentally ill [ 11. Such groups, while similar in overall purpose, may take on a variety of formats based on size, duration, session length, specific content, range of membership (such as nuclear versus extended families, siblings, neighbors) and structure, among other variables [2]. Therefore, making comparisons between groups with regard to relative success or utility is a precarious undertaking. What is functional in one setting may not be generalizable to another. There is also no reason to assume that all psychoeducational programs should focus on similar outcomes. Still, because programs for families of the mentally ill are believed to be useful services, it is worth reviewing the types of outcomes which nurses and other professionals consider to be important, and how group leaders determine whether they are being met. The purpose of this paper is to review existing methods of psychoeducational program evaluation, the rationale for those methods, and finally to present a specific evaluation tool developed by this author. Much emphasis will be placed on the rationale for evaluation, which should always be based on the needs and interests of the group leader rather than a less focused desire to be scientific. The process of instrument development is detailed to provide the reader with a general strategy which may be applied to other program settings. A major theme of this paper is that program evaluation can be a creative process, Ltd.
206
and leaders should be reluctant to adopt instruments used in other psychoeducational programs without at least considering differences in group and organizational settings. Types of knowledge generation
Program evaluations are typically quantitative in nature. In psychoeducational programs, for example, positive outcomes may be measured in units including change scores on a pretest and posttest of knowledge about the effects of psychotropic medications, or the number of contacts which participants initiate outside the group setting with other families of mentally ill persons. However, the science of research has been undergoing a transformation during the past 20 years in recognition of the idea that qualitative methods of knowledge generation are as valid in certain circumstances as those focused on hard data [3]. The program evaluator need not feel compelled to rely only on statistical methods, as these may produce evidence of change among participants which nonetheless misses the essence of that change. Likewise, changes may take place which cannot be captured by statistical methods at all. Particularly in psychoeducational programs, where the numbers of participants are small and the group process generated is as important as particular outcomes, qualitative or case study methods can be quite useful to the professional. There are a number of emerging scientific paradigms, or ways of generating knowledge, but for purposes of this discussion only two will be outlined. The positivist paradigm, or traditional scientific method, assumes that external reality has precision as a system of distinct, observable, interacting variables which have quantity and operate in regular patterns [4]. The goal of science, which is culture and value-free, is to discover those patterns and thus explain phenomena. Evaluation in this sense is a controlled, systematic process of working with variables statistically.
Critics of this method argue that the scientific method does not translate well to the realm of human science. Any breakdown of social phenomena into discrete variables is said to be reductionistic, reflecting the biases of the researcher as much as external reality. Therefore, an interpretive paradigm has gained prominence in the social sciences which assumes that human uniqueness is found in the symbols groups use to define their realities [5]. Inductive inquiry is pursued in natural settings toward the goal of allowing subjects to develop their own outcome themes. From this perspective, the program evaluator attempts to minimize his or her own expectations of group process and outcome and allows the group to set its own priorities. Because interpretive processes will be unique to each situation, outcome themes will be different for each set of individuals participating in a psychoeducational program, even when it is conducted by the same mental health professional in the same agency. The point to be emphasized here is that the psychoeducational program evaluator should not become preoccupied with technique prior to having had group experience, and should not feel compelled to secure evidence of positive outcome within a particular methodological paradigm. Statistical methods facilit: ate the identification of trends and major categories of program impact, but qualitative methods are illuminating with the relative richness of detail they provide. Both methods have been used productively in programs for families of the mentally ill, and it is to examples of these that this narrative now turns. Examples of program evaluations
While psychoeducational programs are varied in format, it is possible to examine and compare how a number of well-known groups have been evaluated. An early group program at Thresholds, a psychosocial rehabilitation center in Chicago, compared families participating in a weekly, 12-session series of
207
meetings with a nonequivalent control group of nonparticipant families [6]. Outcome measures for the group, conducted by several mental health professionals and including the client family member, consisted of parents’ reports of significant gains in attitude restructuring toward their impaired children, and the rate of members’ children who left home during the course of the program. A short questionnaire devised by the group leaders specifically for this pilot program was used to secure parents’ feedback relative to the outcomes. The leaders admitted that their evaluation design was far from conclusive, and that they were looking somewhat nonspecifically for encouraging results utilizing subjective criteria. Another program implemented a family skills-training component along with educational approaches toward the goal of improving family stress management [7]. Single families attended a series of six weekly sessions with mental health agency professionals following the client relative’s hospital discharge. Outcome was measured in terms of rehospitalization rates during a 6-month follow-up period, using a nonequivalent control group of agency clients whose families did not participate in the program. The data were retrieved from client records and family member reports. While 104 families were eventually included in the program evaluation, it must be noted that, as a part of the study, clients in the treatment group were maintained on higher doses of medication. One purely supportive psychoeducational model sought to enhance the ability of families to cope with the stress of caring for a mentally ill member without overtly attempting to change behavior [8]. A single clinician met with a family in their home, only as often as the family requested. The professional worked to aftirm that the family was not responsible for the client member’s mental disorder, that professionals have probably not been sufficiently responsive to families, and that the clinician wanted to offer whatever support
might be helpful. Using a short satisfaction survey given as the visits ended, the programmers sought to investigate whether families were comfortable with the model and if it contributed to a reduction in subsequent hospital days for the disabled family member. The application of experimental methods to program evaluation is generally impractical for agency-based professionals, but these have been utilized by several practitioners with university affiliations. One trio of researchers randomly assigned clients with schizophrenia to four treatment groups, each of which contained some combination of family intervention, social skills training, behavior therapy, and chemotherapy 191. The psychoeducational component of their comprehensive intervention consisted of a day-long survival skills workshop in which groups of families were provided with information and an overview of the social skills training intervention for their client relatives. This 8-h session, in which a number of professionals participated, was followed by single-family sessions every 2-3 weeks for 6 months to 1 year in which information from the workshop was monitored and applied. The program was evaluated after 1 year by comparing rehospitalization rates among the four groups. Unfortunately, the effects of the psychoeducational component of the intervention could not be isolated from the other features. A more specific psychoeducational program was evaluated with an experimental design and both qualitative and quantitative methods by another group of researchers [lo]. They conducted a two-session, in-home educational program for all members of single families and included the member with schizophrenia. The first session focused on the nature of schizophrenia, and the second concentrated on issues of medication as the cornerstone of the community management of schizophrenia. Three professionals participated in the presentation; session length was not reported. The goal of this intervention was simply to educate families, based on the
208
rationale that knowledgeable families will be attuned to the warning signs and ongoing risks of relapse with the disorder of schizophrenia. The authors randomly assigned 36 families into single experimental and control groups of equal size. Control group clients were seen for individual treatment. The evaluation instrument was a two-part questionnaire designed to measure knowledge about schizophrenia. The first part consisted of six open-ended questions about diagnosis, causes of the illness, medications, and therapeutic treatment factors. An independent rater coded responses into categories based on their frequency. The second part included fourteen multiple-choice questions testing knowledge of schizophrenic symptoms, medication and other treatments, side-effects, and the effects of stress and life events on the client member. Mean scores of the responses on this part of the questionnaire were tabulated for group comparisons. Participants were tested with the same instrument immediately after the family sessions, and then 3 and 9 months later. Average scores were compared between the experimental and control groups, but their sizes were too small to permit the use of inferential statistics. The program evaluation methods outlined above utilized a variety of outcome measures and designs, but almost all were applied statistically. Many of them concentrated on client-member measures of symptomatology or rehospitalization, rather than outcomes related to the family caregivers themselves. As will be implied below, program outcomes focused on the caregivers can be more suited to the inclusion of qualitative evaluation approaches than those concerning the client relative. These inductive approaches can be quite valid when carefully applied. Two similar education and support groups for families of the mentally ill have been evaluated qualitatively. The first was a weekly, ten-session program, including approx-
imately eight families in each series, whose clients were treated at an out-patient aftercare clinic [ 111. The purposes of the program were to provide families with information about mental illness, enable them to take more time for themselves, and teach them management techniques for improving client behaviors at home. The evaluation consisted of a built-in program feature in which parents were asked each week by the professional staff to make suggestions about how future meetings or series could be made more beneficial. One of the three ongoing leaders served as a designated recorder and copied down all feedback for later discussion at the regular betweengroup staff meetings. The reasons for disapproval of any family suggestions were discussed at the psychoeducational meetings. The assumptions of these evaluators were that the of member preferences accommodation would result in maximally beneficial groups, and that members would openly provide such feedback. In fact, the program did evolve significantly through this process. The coleaders also compiled family case narratives which were content-analyzed to identify emergent themes in the program for each set of participants. The present author conducts a similar type of group program (nine weekly sessions of 90 min each, including eight to twelve participants) in which the goals of education and the production of greater family support are pursued [ 121. The initial evaluation process involved a pretest, administered by phone by the group leader to determine how much families know about their client relative’s diand treatment inagnosis, medications, volvements. The short survey questionnaire also asked families to describe their major concerns about the client member and what they hoped to gain from the group experience. A posttest survey, given at the final session, asked participants to describe in open-ended fashion anything new they had learned relative to the three areas noted above, any changes they had initiated within their
209
families or social milieus to feel less isolated and more comfortable, and any suggestions for program improvement. This rather straightforward approach was intended to provide the leader with feedback about program content and the process of sharing among members for use in the planning of future series. One shortcoming of this method was that it secured no feedback from families who dropped out of the program without explanation or notice. Some programs, of course, are not evaluated at all with regard to specific outcomes, and while this is generally an undesirable practice, it does not suggest in every case a lack of competence or ability to determine effects. Probably no one in the mental health field has done more to promote the value of the psychoeducational format than Hatfield [13], yet nowhere in her volume on program implementation theory and practice does she discuss criteria for evaluation. Still, her approach is conceptually based on a solid review of the literature, and she devotes more attention to educational theory than any other writer in this area. In her view, psychoeducation is a process that will continue for families long after their participation in her program ends. Hatfield’s emphasis on a long-term process which will utilize input from many sources, professional and otherwise, renders a limited-scope evaluation of little relevance for her. An example of instrument development
The present author’s Family Education and Support Group, noted earlier, is a timelimited program for the adult family members of a mental health agency’s severely mentally disabled clients. The nine sessions, conducted by a social worker, consist of four structured educational presentations alternating with five group discussion and problem solving meetings. After 7 years of continuous operation, with the program offered three times yearly, a decision was made to revise the
evaluation procedure by adding a quantitative component in order to compile some statistical information longitudinally. The process by which the evaluation instrument was balanced with quantitative and qualitative components is detailed here to provide the reader with one example of how the activity can unfold. The question investigated in the current evaluation project is: Do families participating in the Family Education and Support Group have more involvement with supportive individuals and groups outside the nuclear family, and less involvement with their client relative’s activities of daily living, both immediately after and 1 year after completing the program than they did prior to beginning the program? Increased involvement with social support resources, including the client’s mental health treatment providers, indicates an increase in family coping skills and participation in the client’s rehabilitation program. Less involvement with the client relative suggests reduced dependence of the client on the family. An effort is made to determine the effects of the family group, and to investigate whether the acquisition of benefits at the end of each series is maintained after 1 year. There is no intention to generalize results to any other types of psychoeducational programs. Specification of variables
The two dependent variables focused on in this program are (a) the level of independent living skill exhibited by the client relative, and (b) the level of social support secured by the family with the goal of enhancing their coping and management skills with the client relative. Both of these variables derive from the stated goals of the program. Independent living skills. Perhaps the most significant research of the past decade in the area of family interactions as influenced by chronic mental illness has centered on the effects of expressed emotion. Findings suggest
210
that the more face-to-face contact a family and client maintains, the more likely negative emotions will be expressed, and higher stress levels will be experienced by the client relative. Subsequently, there is a higher likelihood of relapse into psychosis and rehospitalization. Thus, if the client’s independent living skills increase, he or she will function less dependently on the family, and expressed emotion will be reduced. In the education component of the program, families are taught methods for enhancing the client’s development of these skills through behavioral management. A constitutive definition of independent living skills for application to the evaluation process was derived by the author from a review of the literature [9,10,13]. It was defined as the ability of a chronically mentally ill client to manage a budget, maintain physical health, keep his or her surroundings clean, acquire basic material resources, maintain supportive relationships with family and friends, engage in regular community activity, and participate regularly in a mental health treatment program. These skills can be possessed in degrees, and do not necessarily imply that the client is living on his or her own. This concept was eventually operationalized in the quantitative portion of the independent living skills inventory as a change score derived from family member perceptions of the client’s skill levels in the seven areas described above, at the end of the program and at a l-year follow-up. Social supports. Another major issue with families of the chronic mentally ill is their tendency toward social isolation and the deleterious effects this has on their ability to cope with stress [ 151. By broadening their range of supports, family members can more effectively pursue their own goals and maintain the energy and motivation to work consistently toward client member’s the rehabilitation. As well as being sources of guidance, social supports provide recreation
for the relief of tension and serve as diversions from focusing too much energy and attention on the client. Based on a second literature review [9,10,16,17], the constitutive definition of social supports in this project was all resources outside the nuclear family, including extended family, friends, co-workers, peer groups, service organizations, and mental health professionals, which provide the family the opportunity to speak freely about the stresses of living with or near a chronically mentally ill relative, and which promote each family member’s relaxation, coping skills, and physical health. Like the first variable, social supports are defined quantitatively as a change score on the social support utilization section of the instrument. Instrumentation The Family Education and Support Group program evaluation instrument, titled as a Member Assessment, is included as an appendix to this paper. It consists primarily of (a) two Likert-type scales, each of which measures characteristics of one of the variables discussed above, and (b) an equal number of open-ended questions intended to acquire more detailed, individualized responses to the scaled items. Three demographic items are included at the bottom of the form, and are formatted in a style conducive to accurate response [ 181. Likert scaling was chosen for the quantitative portion of the evaluation because of its effectiveness in measuring attitudes and affects [ 191. Family members are asked to report their perceptions of the client’s independent living skill level, which tap into feeling responses, and on their own social patterns, which are self-perceptions. Likert scaling was also chosen because Summated scales permit longitudinal comparisons. The instrument is administered to the same individuals three times, and change scores are easily computed as operational measures of each variable.
211
With this measurement format, each respondent receives two scores at each administration. The independent living skills variable, with 14 items and a O-3 range in scoring for each item (negative to positive), contains an overall scoring range of O-42. The scale measuring the variable of social support, with 10 items, has a range of O-30. The two posttest scores can be compared with one another to evaluate the group program’s impact on those variables. The narrative feedback included with each scale item can be content analyzed for additional feedback about program impact on participants. The evaluator organizes respondent comments into categories, which are not determined in advance but are based on emerging themes, and constructs a summary profile of each group. The response formats are the same for both scales. A four-point very rarely/infrequently scaling method was chosen as it permits an item wording format which reduces the possibility of socially desirable responses. That is, while individuals may be reluctant to admit that they have no access to a desirable coping resource (as in agree/disagree scaling), they may admit to utilizing the same resource infrequently. The directions for completing these sections of the instrument include a description of the variable under consideration to orient respondents to its purpose. Group members do not describe themselves in the first scale, so there is little danger of the variable notation prompting certain socially desirable responses. However, identifying the second variable, social supports, by name might elicit such a response set and thus is not included. Scale items are represented as ways of coping with stress, which seems to be a more neutral terminology. Content validity for the scales was established with the assistance of a panel of experts. Initially, 21 items were written to operationalize the independent living skills variable, and 16 were written for the social supports variable. The constitutive definition
of the former variable contained seven distinct components. Three items were written as measures of each component, all of them original and based on discussions in the literature as well as the author’s own clinical experience. The latter concept did not lend itself quite as systematically to item development. However, the 16 items were also original and based on factors included in the constitutive definition of social support. These items were felt to reflect major themes found in the literature review. Next, a document was composed for distribution to a seven-member panel of experts. The panel, consisting of mental health professionals from area agencies with at least 5 years of experience with chronically mentally ill clients and families, was asked to review the variable definitions and measurement items. The items were to be rated as 1, 2, or 3 (low to high) depending on the expert’s opinion about the relevance of each statement’s placement into its conceptual category. Panel members were also asked to comment on the wording of items. The intent was to include the two highest rated items for each of the seven components of the independent living skills variable in the final instrument. However, no statement would be used with a mean rating of less than 2.0. For the social supports variable, the plan was to include all items in the final instrument with a mean rating of 2.0 or above. The results of the content validity exercise were illuminating. For six of the seven components of the first variable, at least two items received ratings of 2.0 or higher and were thus included in the final instrument. Only one statement reflecting the remaining component received a 2.0 rating. A written suggestion by one expert that the second-highest rated statement be reworded was followed, and that item was included in the program evaluation instrument. Its rating had been only one point shy of a 2.0 rating in its original form. The social support items were apparently not as well developed as the former statements. Only 8 of
212
16 received the necessary rating score. Two experts suggested additional items to be included as measures of this variable, and this was done, bringing the total to 10. Clearly, the use of a panel of experts resulted in a significant editing of the instrument. As stated earlier, the narrative comments of group members about each of the scale items are categorized for theme identification. In this process the leader lists each participant’s feedback on a separate sheet by topic in order to construct a group profile. This author develops topics for presentation to the group based on the pretest responses. The narratives also provide more detail about any changes taking place within families over time. Family members tend to note their ongoing problems on the first posttest which helps the leader understand what issues may merit more attention in subsequent groups. One of the demographic items adds to the scope of data collected with the instrument. The item concerning place of residence is a rank-order measure, from most dependent (family) to least dependent type. If at the time of first or second posttest the client relative has moved into another level of housing, the independent living skills variable is reviewed to see if its score changes in the same direction. This demographic indicator is not used at the present time as a measure in the evaluation process, but if it seems to correlate significantly with the first variable it may be used as such in the future. Face validity of the form was examined by distributing draft copies to live former members of the Family Education and Support Group. These persons were asked in writing to comment on the clarity of the survey, the ease of its completion, and the relevance of its content. Several changes in the format and order of items resulted from this exercise. Data collection Program evaluation is to be collected at three stages: during the first meeting of the
Family Education and Support Group (the pretest), at the ninth and final group meeting (posttest), and 1 year after the final meeting, by mail (second posttest). The same instrument is used at all three data collection stages. Change scores and narrative themes are calculated for each respondent between the pretest and first posttest to determine shortterm program impact, and between the first and second posttests to evaluate long-term benefits of the program with regard to the variables under consideration. No problems have been encountered in collecting data during the first two stages. New group members are asked to complete the pretest after their introduction to the program at the first session. Individuals who for some reason miss the initial meeting are asked to complete the form during the short break during session number two, the last meeting to which new members can be admitted. Members complete the posttest at the ninth and final session. Those who miss the final session are phoned by the group leader and asked to till out the evaluation at home, and copies are sent with a stamped return envelope. A time period of 2 weeks is set for the return of mailed forms to minimize the potential effects of history on the respondents. Data collection 1 year later, which has not yet occurred, will be more complicated. During the anniversary month of each group’s termination, the leader will contact former members by phone. He will remind each family of the l-year follow-up that was discussed during the program, and announce that a copy of the member assessment form will be mailed within 2 weeks. The purpose of the follow up will be presented simply - that the agency is interested in learning about the status of each family’s situation relative to the emotionally disabled family member so as to update and improve the program’s content. Former group members have traditionally enjoyed keeping in touch with the agency following their program involvement, so no major problems are anticipated in securing
213
cooperation. One problem may involve former members who have moved. The leader maintains names and addresses on file from past groups, but there may be a significant subject mortality rate at the l-year follow-up. The Member Assessment form will be sent to former members with an accompanying cover letter and a stamped, addressed envelope. Respondents will be given 2 weeks to return the material. If it is not submitted, the leader will again call the individual and ask him or her again to complete and return the form. If still not received within 2 weeks, a second packet will be sent. As a final reminder, a post card will be sent to any nonrespondents another week later. Evaluation data collected from several groups, and covering a period of years, will hopefully become useful in monitoring program usefulness to family members. Summary
Program evaluation is within the capability of every mental health professional. Most practitioners attempt to monitor the quality of the services they provide; if not by formal measurement then through the use of clinical supervision, case consultation, and observations of client behaviors. However, as has been demonstrated in the foregoing examples, a number of adaptable techniques have been developed for psychoeducational program evaluation which can provide objective feedback on outcome attainment. Some of these involve stricter research designs than many practitioners have the inclination or authority to implement. There is, in fact, little incentive in most agencies for clinicians to engage in elaborate evaluation projects. Fortunately there are other instruments and outcome indicators which can be readily incorporated into a program structure. Ant evaluation format should be consistent with the purpose of the activity and the priority program outcomes. Evaluation designed to suit the directives of administrative bodies are
common but can be irrelevant to the concerns of group members, unless the program leader is independently motivated to examine clinical issues as well. All of the evaluations described in this paper have been focused on various outcomes relevant to the program participants. This writer is biased towards those formats which incorporate the qualitative comments of participants in addition to such statistics as client member rehospitalization, as each serve useful but somewhat different functions for the evaluator. A mix of outcome measures can be included in any evaluation design, and formats different from those outlined in this paper are likely to emerge in newer programs. Just as psychoeducation has assumed diverse forms for various target populations, there is room for innovation in the methods of its evaluation. References 1 2 3
4
5 6
Hatfield AB, Lefley HP eds: Families of the Mentally III: Coping and Adaptation. New York: Guilford, 1987. McFarlane WR ed: Family Therapy in Schizophrenia. New York: Guilford, 1983. Popkewitz T : Paradigm and Ideology in Educational Research: The Social Functions of the Intellectual. New York: The Falmer Press, 1984. Ary D, Jacobs LC, Razavieh A: Introduction to Research in Education, 3rd edition. New York: Holt, Rinehart, and Winston 1985. Lincoln YS, Guba EG: Naturalistic Inquiry. Newbury Park, CA: Sage, 1985. Dincin J et al: Restructuring parental attitudes: working with parents of the adult mentally ill. Schizophrenia Bulletin 1978; 4: 597-608. Goldstein MJ, Kopeikin HS: Short- and long-term effects of combining drugs and family therapy. In: Goldstein MJ ed. New Developments in Intervention with Families of Schizophrenics. San Francisco: Jossey-Bass, 1981. Byalin K et al: Family intervention with treatmentrefractory schizophrenics. 20th International Congress of Applied Psychology, Edinburgh, 1982. Anderson CM et al: Schizophrenia and the Family. New York: Guildford, 1986. Falloon IR et al: Family Care of Schizophrenia. New York: Guilford, 1984. McLean CM et al: Group treatment for parents of the adult mentally ill. Hosp Commun Psychiatry 1982; 33: X4-569. Walsh J: Psychoeducational program evaluation: one practical method. J Psychosocial Nurs Mental Health Services 1987; 25: 25-31.
214
13 14
15 16 17 18
Hatfield AB: Family Education in Mental Illness. New York: Guilford, 1990. Koenigsburg A, Handley R: Expressed emotion: from predictive index to clinical construct. Am J Psychiatry
19
Mueller DJ: Measuring Teacher’s College, 1986.
Social
1986; 143: 1361-1373. Vine P: Families in Pain. New York: Pantheon, 1982. Kantor JS ed: Clinical Issues in Treating the Chronic Mentally III. San Francisco: Jossey-Bass, 1985.
Correspondence
Families of the Mentally III Collective: Families Helping Families. New York: W.W. Norton, 1986. Sudman S, Bradburn NM: Asking Questions. San Francisco: Jossey-Bass, 1982.
J. Walsh North Community Counseling Centers 4897 Karl Road Columbus, OH 43229, USA
Attitudes.
New
York:
to:
Appendix: The family education and support group member assessment Name:
Date:
In this first series of statements, please assess your emotionally disabled family member’s independent living skills. Please remember that these can be possessed in degrees and do not necessarily indicate that your relative is living on his or her own at the present time. As you consider each item, use the past month as a time frame in deciding upon your response. Underline the response which most closely indicates your assessment of each statement, as demonstrated by the example. Example:
My family member is a friendly VERY RARELY OCCASIONALLY person. FREQUENTLY RARELY This respondent feels that the family member is sometimes friendly, not as friendly as he or she might be. Now, please proceed to complete the scale below, and elaborate on each of your responses in the space beneath each item. My emotionally disabled family member: 1. Uses money to pay his/her personal expenses in a reasonable way. Explain:
2. Takes care of housekeeping without being reminded. Explain:
3. Maintains Explain:
an adequately
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
tasks
balanced
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
diet.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
215
4. Arranges promptly Explain:.
for transportation when needed.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
5. Spends time with friends outside the home. Explain:
6. Engages in recreational outside the home. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
activity
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
7. Makes an effort to secure income from a source outside the family. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
8. Keeps his/her
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
clothing
clean.
Explain:
9. Maintains reasonable expectations of the family for companionship. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
10. Takes the initiative or maintain a job. Explain:
to achieve
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
as prescribed.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
11. Takes medication Explain:
216
12. Has a physician whom he/she visits when physically ill. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
13. Purchases his/her own clothing and household supplies. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
14. Meets at least monthly with a mental health counselor or case worker. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Next, rather than thinking about your family member, please consider your own situation. Listed below is a series of statements reflecting various types of personal relationships and activities. These refer to different ways in which persons may cope with the stresses associated with having a disabled family member. There are no right or wrong responses to these items, since all people cope with stress differently. Next to the statements below, please underline the response which best describes your frequency of participation. Again, elaborate on your responses in the space below each item. Example:
I exercise to work off tension.
This respondent
hardly
OCCASIONALLY RARELY
if at all. Please go on now to complete
the scale items below.
hobbies
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
in at least one social group.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
1. I engage in personal outside the home. Explain:
2. I participate community Explain:
ever gets exercise,
VERY RARELY FREQUENTLY
3. I have contact with my family member’s mental health service provider. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
4. I spend time with friends for purposes unrelated to my family problems. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
5. I can devote uninterrupted time to my job or my own routine daily activities (if unemployed outside the home). Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
6. I visit a mental health professional for help with my own problems. Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
I confide in the following people about my family problems: 7. Neighbors.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Explain:
8. Extended family members (aunts, cousins, grandparents, etc). Explain:
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
9. My physician.
VERY RARELY OCCASIONALLY FREOUENTLY RARELY
Explain:
10. My pastor.
VERY RARELY OCCASIONALLY FREQUENTLY RARELY
Explain:
The emotionally disabled member in my family is my: SON DAUGHTER MOTHER FATHER
BROTHER SISTER SPOUSE OTHER (please specify:)
218
He/she lives in: MY HOUSE OR APARTMENT A GROUP HOME FOR EMOTIONALLY DISABLED PERSONS AN AGENCY-MANAGED OR SUPERVISED APARTMENT HIS OR HER OWN HOUSE OR APARTMENT OTHER (please specify:) years old and has been in treatment at this or other mental health agencies or He/she is now ~ years. psychiatric hospitals for approximately _ Other comments?
THANK YOU VERY MUCH FOR YOUR COOPERATION