- Email: [email protected]
Teaching evaluation through the practicum experience
54
• Make a concerted effort to develop interpersonal, interviewing, and group facilitationskills • Include inslitutional development and evalualion sustainability on your list of priorilies.
REFERENCES Patton, M. Q. (1986). Utilization-focusedevaluation (2nd ed.) . Beverly llills, CA: Sage. Patton, M. Q. (1987). Creative evaluation. Newbury Park, CA: Sage . Santo Pietro, D. (t983). Evaluationsourcebookfor private and voluntary organizations. New York: American Council of Voluntary Agencies for Foreign Service.
TEACHING EVALUATION THROUGH THE PRACTICUM EXPERIENCE David P. l\Ioxle,}' Wayne Slate University Richard J. Visingardi Michigan State University
Teaching and learning program evaluation are challenging tasks for educators and students. At the Developmental Disabilities Institute located at Wayne State University in Detroit, Michigan, learning to evaluate service delivery to developmentally disabled people is a core competency area for the Institute's graduate social work interns. The mission of the Institute is to provide resources such as technical assistance, applied research, and program evaluation to community agencies serving people with developmental disabilities. The evaluation practicum has emerged within the Institute as a major vehicle for teaching social work students the many realities and demands of conducting evaluations within actual agency settings, An agency with evaluation needs is identified, and one student or a group of students initiates an evaluation, beginning with problem definition and question formulation and moving through all major steps of the project. A social work faculty member serves as preceptor for the completion of the project.
55
This article summarizes the major stages one student moved through in completion of an evaluation practicum within a community-based program providing day services to people with severe developmental disabilities, and offers suggestions for others wanting to set up practicum experiences for students.
STUDENT EXPERIENCES IN THE PRACTICUl\l The site for the student's practicum was a program of a large rehabilitation agency located in the Detroit metropolitan area. The program served approximately 60 clients with severe medical and/or behavioral problems. The program was relatively new and represented a new service delivery direction for the parent agency. Se veral factors guided the selection of the practicum site: (1) The program was in a stage of development/implementation in which process and formative evaluations were critical to guiding its continued growth (Moxley, 1987; Tripodi, Fellin , & Epstein, 1978); (2) the agency was motivated to complete formative evaluation of the program because it was approaching an external accreditation review and was seeking input about the program 's functioning; (3) program staff were very motivated to obtain critical feedback about their habilitation efforts with clients ; and (4) the agency was seeking linkages with the university as a means of increasing technical assistance in program evaluation. These factors created an ideal learning environment for a student seeking to gain evaluation skills through practical experience. The factors favored a "controlled situation" in which the defensiveness and resistance of the host agency were minimal. Process ofthe practicum. The student moved through four distinct stages in the process of completing the evaluation practicum. These stages required a time commitment on the part of the student and the preceptor of approximately six months. The student invested about 15 hours per week during this period, while the preceptor made a commitment of approximately 175 hours. Each stage is discussed below. Entry into the program. Individuals at the practicum site had very little experience with evaluation and did not know what to expect of the process. Thus the student and the preceptor entered the evaluation situation by meeting with the program director and her immediate supervisor. Three meetings were held, during which the student negotiated the purpose and focus of the evaluation and the final product of the evaluation. To accomplish this work, the preceptor coached the student in semistructured interviewing techniques, and met with the student prior to each meeting to frame the purpose and goals of the negotiation process.
56
Development of evaluation questions and selection of methodology. The student designed and wrote an evaluation prospectussummarizingthe overall goals and strategy of the evaluation. This document served as thecontract between the Institute and the program. It delineated the roles and responsibilities of each of the major parties and identified the resources needed to complete the project. To operationalize the focus of the evaluation-that is, to define relevant ends and information needs-the student held several meetings with major internal and external program stakeholders. In addition, the student spent some 40 hoursin the program as a participantobserver. At the completionof this stage, in consultation with the preceptor and through interviews with staff members and the program administrator, the student defined specific evaluation questions, identified an audience of five groups of clients to be observed, and chose a behavioral observation strategy for obtainingdata on a representative sample of clients served by the program. Then, working with the preceptor, the student designed a behavioral recording instrument to capture in-vivo data on the habilitation activitiesof staff, developed a methodology for comparing these data to the habilitation plans found in the clients' clinical files, randomly selected five clients from each group, and systematically observed these clients over several weeks using a timesampling procedure. Data analysis ami staff feedback. Using a microcomputer statistical package, thestudentanalyzedthe dataat plannedintervalsand heldfeedback sessions with both line and administrative staff. Emphasis was placed on describing the habilitation activities actually being undertaken by staff and the compatibility of theseactivitieswith the individualized habilitation plans of clients. The studentwas able to identifygroups of clients who were not receiving adequate levels of habilitation. During these meetings the student gained experiencein responding tostaff questionsand in leadingdiscussionsof how the implementation of theprogramcould be changedto better meet the needs of clients. An outcome of these meetings was the identification of recommendations and issues for the final summative report. Report development and termination. Several drafts of the report were reviewed with program staff. Report recommendations addressed program weaknessesidentifiedthrough the evaluation,such as the organization of the habilitation process, which was thought to be responsible for a number of clients not receiving adequate or consistent habilitation. Final summative meetingswere held with line staffand administratorsto identify a plan of action for utilization of the information identified in the final project report.
57
OBSERVATIONS FOR OTHERS SETTING UP PRACTICA
The program evaluation practicum in developmental disabilities is a very intensive experience. We have found four supports to be important in ensuring the successof the experience. First, the preceptor mustbe very accessible to the student and willing to serve in the roles of consultant,methods teacher, and problem solver, and sometimes as mediator between the student and program staff. Second, stakeholders at the practicum site must be willing to work with the student and must recognize that since the student is learning new skills andcapacities, some mistakesmay be made.Third,the host agency must be willing to contribute tangible resources to the success of the practicum. In the case study reported here, it was important for the agency to make key supervisors available to the student, and for these supervisors to review instruments,procedures, and reports on a timely basis, and to provide limited in-kind resources such as copying, access to a telephone, and secretarial assistance. It is important for the preceptor to monitor the practicum site and to ensure that a clear contract is in place that covers the level of effort expectedof the agency before thestudent actuallybegins the project. Fourth, it is helpful for students to be taking formal courses in research, evaluation, and administration while completing the practicum. Such coursework helps the student gain a better grasp of methodological, analytical, and organizationalcontent useful in completing the project. Locating environments for students usually occurs through the professional network of the preceptor. Thus preceptors can identify potential practicum sites through the network of agencies with whom they have established technical assistance, consultation, or advisory relationships. However, in recruiting practicum sites preceptors may want to follow three suggestions: (1) Base the practicum on an identified evaluation need of the agency. This requires the preceptor to have some familiarity with the agency and to be willing to spend some time with decision makers in defining these needs. (2) Assure the agency that adequate supervision and technical skills arc availabte to the student. Preceptors should not set up the expectation that the agency will provide these resources to the student. (3) Frame the practicum in a "developmental" context. Agencies arc most cooperative when the preceptor and student present the practicum as a learning experience for the student and as a growth opportunity for the agency.
The program evaluation practicum is a useful vehicle for encouraging students to learn the pragmatic, interpersonal, and technical dimensions of evaluation. Such experience may also lead to the development of stronger
58
and more informed practitioners within the field of developmental disabilities.
REFERENCES Moxley, D. (1987, March). Program planning and evaluation in the dynamic environment of community mental health. Plenary paper presented at the Conference on Program Planning and Evaluation, Detroit-Wayne County Community Mental Health Board, Detroit, MI. Tripodi, T., Fellin, P., & Epstein, I. (1978). Differential program evaluation. Itasca, IL: F. E. Peacock.
TEACHING EVALUATION THROUGH PAID CONTRACTUAL ARRANGEMENTS J. Nicholls Eastmond, Jr. WaIter Saunders Utah State University Duane Merrell Emery County High School, Castledale, Utah
Many of us now teaching evaluation classes at the university level had our beginnings in the field in actually "doing evaluations," as consultants, interns, or other workers. Some of us harbor suspicions that learning the trade in the classroom leads to a very limited set of skills, and that conducting an actual evaluation represents a powerful, if not/he most effective, way to learn the skills so important in program evaluation. During three separate quarters of teaching a graduate-level course titled "Alternative Evaluation Methodologies," I have taken on actual evaluation work as the class project, for example: (1) for a private consulting firm from Salt Lake City for a statewide project of inservice for teachers, the Utah Skills Project (1983), and (2) an evaluation of a National Science Foundation (NSF) program, the Learning Cycle Project, headquartered at Utah State University. During winter quarter 1988, the class was contracted to do instrument design and fieldwork for data collection and analysis. During summer quarter 1988, the class conducted a metaevaluation of the winter quarter class's work.
• Make a concerted effort to develop interpersonal, interviewing, and group facilitationskills • Include inslitutional development and evalualion sustainability on your list of priorilies.
REFERENCES Patton, M. Q. (1986). Utilization-focusedevaluation (2nd ed.) . Beverly llills, CA: Sage. Patton, M. Q. (1987). Creative evaluation. Newbury Park, CA: Sage . Santo Pietro, D. (t983). Evaluationsourcebookfor private and voluntary organizations. New York: American Council of Voluntary Agencies for Foreign Service.
TEACHING EVALUATION THROUGH THE PRACTICUM EXPERIENCE David P. l\Ioxle,}' Wayne Slate University Richard J. Visingardi Michigan State University
Teaching and learning program evaluation are challenging tasks for educators and students. At the Developmental Disabilities Institute located at Wayne State University in Detroit, Michigan, learning to evaluate service delivery to developmentally disabled people is a core competency area for the Institute's graduate social work interns. The mission of the Institute is to provide resources such as technical assistance, applied research, and program evaluation to community agencies serving people with developmental disabilities. The evaluation practicum has emerged within the Institute as a major vehicle for teaching social work students the many realities and demands of conducting evaluations within actual agency settings, An agency with evaluation needs is identified, and one student or a group of students initiates an evaluation, beginning with problem definition and question formulation and moving through all major steps of the project. A social work faculty member serves as preceptor for the completion of the project.
55
This article summarizes the major stages one student moved through in completion of an evaluation practicum within a community-based program providing day services to people with severe developmental disabilities, and offers suggestions for others wanting to set up practicum experiences for students.
STUDENT EXPERIENCES IN THE PRACTICUl\l The site for the student's practicum was a program of a large rehabilitation agency located in the Detroit metropolitan area. The program served approximately 60 clients with severe medical and/or behavioral problems. The program was relatively new and represented a new service delivery direction for the parent agency. Se veral factors guided the selection of the practicum site: (1) The program was in a stage of development/implementation in which process and formative evaluations were critical to guiding its continued growth (Moxley, 1987; Tripodi, Fellin , & Epstein, 1978); (2) the agency was motivated to complete formative evaluation of the program because it was approaching an external accreditation review and was seeking input about the program 's functioning; (3) program staff were very motivated to obtain critical feedback about their habilitation efforts with clients ; and (4) the agency was seeking linkages with the university as a means of increasing technical assistance in program evaluation. These factors created an ideal learning environment for a student seeking to gain evaluation skills through practical experience. The factors favored a "controlled situation" in which the defensiveness and resistance of the host agency were minimal. Process ofthe practicum. The student moved through four distinct stages in the process of completing the evaluation practicum. These stages required a time commitment on the part of the student and the preceptor of approximately six months. The student invested about 15 hours per week during this period, while the preceptor made a commitment of approximately 175 hours. Each stage is discussed below. Entry into the program. Individuals at the practicum site had very little experience with evaluation and did not know what to expect of the process. Thus the student and the preceptor entered the evaluation situation by meeting with the program director and her immediate supervisor. Three meetings were held, during which the student negotiated the purpose and focus of the evaluation and the final product of the evaluation. To accomplish this work, the preceptor coached the student in semistructured interviewing techniques, and met with the student prior to each meeting to frame the purpose and goals of the negotiation process.
56
Development of evaluation questions and selection of methodology. The student designed and wrote an evaluation prospectussummarizingthe overall goals and strategy of the evaluation. This document served as thecontract between the Institute and the program. It delineated the roles and responsibilities of each of the major parties and identified the resources needed to complete the project. To operationalize the focus of the evaluation-that is, to define relevant ends and information needs-the student held several meetings with major internal and external program stakeholders. In addition, the student spent some 40 hoursin the program as a participantobserver. At the completionof this stage, in consultation with the preceptor and through interviews with staff members and the program administrator, the student defined specific evaluation questions, identified an audience of five groups of clients to be observed, and chose a behavioral observation strategy for obtainingdata on a representative sample of clients served by the program. Then, working with the preceptor, the student designed a behavioral recording instrument to capture in-vivo data on the habilitation activitiesof staff, developed a methodology for comparing these data to the habilitation plans found in the clients' clinical files, randomly selected five clients from each group, and systematically observed these clients over several weeks using a timesampling procedure. Data analysis ami staff feedback. Using a microcomputer statistical package, thestudentanalyzedthe dataat plannedintervalsand heldfeedback sessions with both line and administrative staff. Emphasis was placed on describing the habilitation activities actually being undertaken by staff and the compatibility of theseactivitieswith the individualized habilitation plans of clients. The studentwas able to identifygroups of clients who were not receiving adequate levels of habilitation. During these meetings the student gained experiencein responding tostaff questionsand in leadingdiscussionsof how the implementation of theprogramcould be changedto better meet the needs of clients. An outcome of these meetings was the identification of recommendations and issues for the final summative report. Report development and termination. Several drafts of the report were reviewed with program staff. Report recommendations addressed program weaknessesidentifiedthrough the evaluation,such as the organization of the habilitation process, which was thought to be responsible for a number of clients not receiving adequate or consistent habilitation. Final summative meetingswere held with line staffand administratorsto identify a plan of action for utilization of the information identified in the final project report.
57
OBSERVATIONS FOR OTHERS SETTING UP PRACTICA
The program evaluation practicum in developmental disabilities is a very intensive experience. We have found four supports to be important in ensuring the successof the experience. First, the preceptor mustbe very accessible to the student and willing to serve in the roles of consultant,methods teacher, and problem solver, and sometimes as mediator between the student and program staff. Second, stakeholders at the practicum site must be willing to work with the student and must recognize that since the student is learning new skills andcapacities, some mistakesmay be made.Third,the host agency must be willing to contribute tangible resources to the success of the practicum. In the case study reported here, it was important for the agency to make key supervisors available to the student, and for these supervisors to review instruments,procedures, and reports on a timely basis, and to provide limited in-kind resources such as copying, access to a telephone, and secretarial assistance. It is important for the preceptor to monitor the practicum site and to ensure that a clear contract is in place that covers the level of effort expectedof the agency before thestudent actuallybegins the project. Fourth, it is helpful for students to be taking formal courses in research, evaluation, and administration while completing the practicum. Such coursework helps the student gain a better grasp of methodological, analytical, and organizationalcontent useful in completing the project. Locating environments for students usually occurs through the professional network of the preceptor. Thus preceptors can identify potential practicum sites through the network of agencies with whom they have established technical assistance, consultation, or advisory relationships. However, in recruiting practicum sites preceptors may want to follow three suggestions: (1) Base the practicum on an identified evaluation need of the agency. This requires the preceptor to have some familiarity with the agency and to be willing to spend some time with decision makers in defining these needs. (2) Assure the agency that adequate supervision and technical skills arc availabte to the student. Preceptors should not set up the expectation that the agency will provide these resources to the student. (3) Frame the practicum in a "developmental" context. Agencies arc most cooperative when the preceptor and student present the practicum as a learning experience for the student and as a growth opportunity for the agency.
The program evaluation practicum is a useful vehicle for encouraging students to learn the pragmatic, interpersonal, and technical dimensions of evaluation. Such experience may also lead to the development of stronger
58
and more informed practitioners within the field of developmental disabilities.
REFERENCES Moxley, D. (1987, March). Program planning and evaluation in the dynamic environment of community mental health. Plenary paper presented at the Conference on Program Planning and Evaluation, Detroit-Wayne County Community Mental Health Board, Detroit, MI. Tripodi, T., Fellin, P., & Epstein, I. (1978). Differential program evaluation. Itasca, IL: F. E. Peacock.
TEACHING EVALUATION THROUGH PAID CONTRACTUAL ARRANGEMENTS J. Nicholls Eastmond, Jr. WaIter Saunders Utah State University Duane Merrell Emery County High School, Castledale, Utah
Many of us now teaching evaluation classes at the university level had our beginnings in the field in actually "doing evaluations," as consultants, interns, or other workers. Some of us harbor suspicions that learning the trade in the classroom leads to a very limited set of skills, and that conducting an actual evaluation represents a powerful, if not/he most effective, way to learn the skills so important in program evaluation. During three separate quarters of teaching a graduate-level course titled "Alternative Evaluation Methodologies," I have taken on actual evaluation work as the class project, for example: (1) for a private consulting firm from Salt Lake City for a statewide project of inservice for teachers, the Utah Skills Project (1983), and (2) an evaluation of a National Science Foundation (NSF) program, the Learning Cycle Project, headquartered at Utah State University. During winter quarter 1988, the class was contracted to do instrument design and fieldwork for data collection and analysis. During summer quarter 1988, the class conducted a metaevaluation of the winter quarter class's work.