- Email: [email protected]
“Doing” evaluation— A student's perspective
"Doing" EvaluationA Student's Perspective John C. Ory Sandra C. Leister University of Illinois at Urbana-Champaign
For several years we have required students in our department's Evaluation Methods course to complete an evaluation project. The purpose of the project is to have students "do" an evaluation from client negotiations through the presentation of final results. Students submit a final evaluation report both to the client and to the course instructor. They are also asked to attach to the instructor's copy of the final report a comment sheet wherein they can clarify their actions and experiences, e.g., client preferences or restriction, politics of the setting, difficulties in data collection. The purpose of this note to Evaluation Practice is to share with other evaluators the comment sheet of one of our students. Why? First the student's comments may encourage those who teach evaluation to assign class projects that require "hands-on" evaluation experiences. Second, I believethe student's concerns, apprehensions, and reservations about evaluation work remind us of our reasons for teaching evaluation. Finally, on those gray days when nobody seems to be reading our evaluation reports or appreciating our efforts, we can read these comments and remember why we enjoy "doing evaluation."
STUDENT COMMENT:
I had some real concerns with my lack of expertise in the area of evaluation. Although I have worked on survey, marketing, needs assessment, and evaluation projects, I have never done an entire program evaluation by myself. My goal in reading through your stack of past evaluations was twofold: to get an overview of what constitutes a good evaluation and also to determine an appropriate format for mine. It quickly became obvious to me that there is no prescribed, customary way to do an evaluation; instead there seem to be only rather 12
13
loose sorts of guidelines. This freedom from constraints was both exciting and intimidating to me. In my previous life (before doctoral work) this was what I was accustomed to, and I enjoyed the freedom of working without a lot of imposed structure. However, being in the counseling psychology program has made me change the way I approach things: Everything created is done in APA style with rigid requirements and specific format-from abstract to margins to conclusions. As I started this project, I felt a little intimidated leaving the security of these restrictions. I now see that the client, the situation, the purpose, the participants, the philosophy of the evaluator, and the type of program all affect the decisions involved in designing and implementing an evaluation. In fact, now it seems imperative to me that the design of the project emerge from these factors rather than being imposed on them. Perhaps the greatest uneasiness I had was in experiencing the conflicts among the various aspects of my professional training as educator, counselor, and evaluator. I was asked to undertake this project because I had some experience in each of these, and it was assumed that I could profitably draw on the training of all three . To a great extent this was true. However, there was a degree of tension within me: The contrasts of the directiveness of a teacher and the determination of value by an evaluator and the unconditional acceptance of a counselor obviously do not provide for a perfect fit. I don't suppose this confused self-concept is unusual and may in fact be parallel to the professional identity crisis frequently noted in the field of evaluation. Perhaps this is why I liked House's definition of the evaluator as the "wise counselor" so much. It seemed generic enough to cover many options and stressed the areas with which I most identify. My counseling training was particularly helpful to my becoming aware of some negative dynamics in the evaluation situation. Although I was assured by some that my client was "thrilled" to have my help and was looking forward to the project, I quickly became aware of the suspicion and defensiveness that was an undercurrent in my preliminary relationship with him . His apprehension was understandable as I was somewhat imposed on him by his department's administration rather than him seeking me out. I do think my counseling background helped in some discussions about our relationship and shared goals, which redu ced his fears and built a strong working relationship. Initially, I thought that I would have a real strength in being an external evaluator, because there wouldn't be any preconceived notions
14
or bias. As I became more involved in the project, I realized this can never be true. We all have opinions and experiences that, at least to some degree, color our perceptions. I had some ideas about what kind of people work in this content area, what the field is like, etc. The more involved I became in the evaluation, the more involved I became with the people-presenters and participants. I can easily understand now why most evaluations are positive; nobody wants to be a bad guy. This ties in with the concern I have about political considerations and the motivation for doing any evaluation. How are the results going to be used? What is going to be the effect of either a positive or negative report on programs, products, and most importantly the people involved? I am not sure that in some situations I want the kind of responsibility that goes along with determining value and utility. When an evaluation is going to have a great positive or negative impact on people, I think I'd prefer to work as part of an evaluation team rather than alone. A situation seen through several sets of eyes, hopefully, would produce more reliable and valid findings and would certainly provide an opportunity for confirming views. In retrospect, I really enjoyed my evaluation. Obviously there is a vast range of what can be the target and outcomes of an evaluation project, and I am most impressed by the sensitivity of such work, especially when crucial results-both positive and negative-can be the outcome. I have a new appreciation and a degree of apprehension for what a tremendous responsibility it can be.
For several years we have required students in our department's Evaluation Methods course to complete an evaluation project. The purpose of the project is to have students "do" an evaluation from client negotiations through the presentation of final results. Students submit a final evaluation report both to the client and to the course instructor. They are also asked to attach to the instructor's copy of the final report a comment sheet wherein they can clarify their actions and experiences, e.g., client preferences or restriction, politics of the setting, difficulties in data collection. The purpose of this note to Evaluation Practice is to share with other evaluators the comment sheet of one of our students. Why? First the student's comments may encourage those who teach evaluation to assign class projects that require "hands-on" evaluation experiences. Second, I believethe student's concerns, apprehensions, and reservations about evaluation work remind us of our reasons for teaching evaluation. Finally, on those gray days when nobody seems to be reading our evaluation reports or appreciating our efforts, we can read these comments and remember why we enjoy "doing evaluation."
STUDENT COMMENT:
I had some real concerns with my lack of expertise in the area of evaluation. Although I have worked on survey, marketing, needs assessment, and evaluation projects, I have never done an entire program evaluation by myself. My goal in reading through your stack of past evaluations was twofold: to get an overview of what constitutes a good evaluation and also to determine an appropriate format for mine. It quickly became obvious to me that there is no prescribed, customary way to do an evaluation; instead there seem to be only rather 12
13
loose sorts of guidelines. This freedom from constraints was both exciting and intimidating to me. In my previous life (before doctoral work) this was what I was accustomed to, and I enjoyed the freedom of working without a lot of imposed structure. However, being in the counseling psychology program has made me change the way I approach things: Everything created is done in APA style with rigid requirements and specific format-from abstract to margins to conclusions. As I started this project, I felt a little intimidated leaving the security of these restrictions. I now see that the client, the situation, the purpose, the participants, the philosophy of the evaluator, and the type of program all affect the decisions involved in designing and implementing an evaluation. In fact, now it seems imperative to me that the design of the project emerge from these factors rather than being imposed on them. Perhaps the greatest uneasiness I had was in experiencing the conflicts among the various aspects of my professional training as educator, counselor, and evaluator. I was asked to undertake this project because I had some experience in each of these, and it was assumed that I could profitably draw on the training of all three . To a great extent this was true. However, there was a degree of tension within me: The contrasts of the directiveness of a teacher and the determination of value by an evaluator and the unconditional acceptance of a counselor obviously do not provide for a perfect fit. I don't suppose this confused self-concept is unusual and may in fact be parallel to the professional identity crisis frequently noted in the field of evaluation. Perhaps this is why I liked House's definition of the evaluator as the "wise counselor" so much. It seemed generic enough to cover many options and stressed the areas with which I most identify. My counseling training was particularly helpful to my becoming aware of some negative dynamics in the evaluation situation. Although I was assured by some that my client was "thrilled" to have my help and was looking forward to the project, I quickly became aware of the suspicion and defensiveness that was an undercurrent in my preliminary relationship with him . His apprehension was understandable as I was somewhat imposed on him by his department's administration rather than him seeking me out. I do think my counseling background helped in some discussions about our relationship and shared goals, which redu ced his fears and built a strong working relationship. Initially, I thought that I would have a real strength in being an external evaluator, because there wouldn't be any preconceived notions
14
or bias. As I became more involved in the project, I realized this can never be true. We all have opinions and experiences that, at least to some degree, color our perceptions. I had some ideas about what kind of people work in this content area, what the field is like, etc. The more involved I became in the evaluation, the more involved I became with the people-presenters and participants. I can easily understand now why most evaluations are positive; nobody wants to be a bad guy. This ties in with the concern I have about political considerations and the motivation for doing any evaluation. How are the results going to be used? What is going to be the effect of either a positive or negative report on programs, products, and most importantly the people involved? I am not sure that in some situations I want the kind of responsibility that goes along with determining value and utility. When an evaluation is going to have a great positive or negative impact on people, I think I'd prefer to work as part of an evaluation team rather than alone. A situation seen through several sets of eyes, hopefully, would produce more reliable and valid findings and would certainly provide an opportunity for confirming views. In retrospect, I really enjoyed my evaluation. Obviously there is a vast range of what can be the target and outcomes of an evaluation project, and I am most impressed by the sensitivity of such work, especially when crucial results-both positive and negative-can be the outcome. I have a new appreciation and a degree of apprehension for what a tremendous responsibility it can be.