Evaluation consultant's topical interest group

Reports from TIGs

Evaluation Consultant's Topical Interest Group MANAGEMENT OF EXTERNAL EVALUATION CONTRACTS BY INDEPENDENT CONSULTANTS: A CONTEXT-DEPENDENT ISSUE? Tara D. Knott Memphis, Tennessee

As an independent evaluation consultant, the management of summative evaluations for repeat customers has become not only an interesting issue but perhaps one that has important implications of more general interest. This professional concern held by independent evaluators represents the initial offering in this section, and is presented for the consideration of the members of the Independent Consultation TIG and the AEA readership at large. Responses and feasible alternative solutions are encouraged and should be sent to me at the address given a the end of the article.

REPEAT CONTRACTS AND LOSS OF EXTERNALITY To state the obvious, in order to establish and maintain any small business, customers are required. Repeat customers are most desirable 79

80

because repeat business is typically an indication that the customer was pleased originally and that the work was useful for programmatic decision making. For a small, independent evaluation firm, repeat business not only enhances self-esteem, it is necessary for survival. However, the independent provision of repetitive evaluations also has its problems regarding the selection of specific contracts for summative evaluations. Much of my independent evaluation work is summative in nature, and many of my clients tell me they contacted us because they want an external, objective evaluation. Note that the contractor equates summative evaluation with objectivity and externality. Although I agree conceptually with those who caution against claiming externality when we are paid by the program sponsor, this situation may be problematic for those of us who practice independently, especially in the context of the private sector. How can one maintain and develop a business without taking repeat business? Let us assume that an independent evaluator receives an original contract for an external, summative evaluation to determine and substantiate program results. Because this experience was a good one in terms of information delivered on time in a useful form for decision making, the company contacts the evaluator a second time two years later to conduct another, similar evaluation of the same program or to expand the original evaluation in another part of the country. The contractor states that he or she called you because you are external to the company and, therefore, objective. What are the independent evaluator's professional obligations with regard to this contract? Can the evaluator be considered truly external at this point? He or she is already familiar with the program and the program sponsor and will, once again, be paid by the program sponsor. What are his or her obligations to the contractor concerning explanations of externality? Although the evaluator realizes that a good summative evaluation can be designed and implemented on an internal or external basis, the potential contractor does not. Is it, then, necessary for the evaluator to identify this issue for the contractor and to discuss its implications with him or her? Consider the evaluator who is retained by a company on a long-term basis-one of the best possible work situations in this evaluator's opinion. Some of his or her summative work will surely be used for marketing purposes at least in the private sector. What is his or her position regarding externality? Does one 's position change based on the contractual arrangements? At some point the evaluator's and, perhaps,

81

contractor's concern for professionally appropriate practices will be an issue. In addition, corporate competitors may begin to question evaluation results from studies conducted by an evaluator who has a long-term relationship with a given company, especially if results are consistently positive. Is the management of external contracts truly a serious professional issue for independent evaluators? Do some of these professional concerns, in fact, change depending on the context in which evaluation is practiced? How far can one stray from the Standards and other professionally sanctioned suggestions and retain one's integrity? These are tough questions. Let us assume that the issue of externality is an important one for independent evaluators and that it should be seriously considered. The following represents this evaluator's thoughts about this matter.

INTERNAL EVALUATION One way to ensure that the work of independent evaluators is professionally sound and free of even the suggestion of biased conduct is to accept repeat contracts only for formative evaluation. This approach neatly removes the independent evaluator from doubt regarding bias in claims of program success. It also removes him or her from the practice of"summative" evaluation and, perhaps, from securing several possible contracts. This is unfortunate because summative or "pay-off' evaluation has been claimed to be the more important evaluation activity and one that I believe must be done. Happily, it is also frequently requested from the evaluator's clients. The problem here is that it is difficult enough to sell* evaluation generally without having to convince potential customers that a perfectly valid summative evaluation can be conducted on an internal basis. Consider the fact that if I or a colleague who is well trained and experienced in evaluation do not accept ajob, the contractor may hire a "good ole boy or girl" from the same industry. He orshe may fall into the field expert traps of having the same biases as the original contractor; and/ or may play the "you scratch my back and I'll scratch yours"game. The unfortunate result of this situation may be that evaluation, once ·Yes, Virginia, independent evaluators use th e word "sell" overtly.

82

again, sinks into near invisibility in deference to management consultants, because evaluators are "eggheads" who are rife with professional problems and difficult to deal with. At the 1985 AEA conference, we heard Michael Scriven suggest that we are diminishing our own sphere of practice and usefulness by our refusal to become involved in unfamiliar, potentially dangerous situations such as the evaluation of program personnel. Is our reticence in accepting repeat summative evaluations a manifestation of this same attitude?

EXTERNAL EVALUATION CONTRACTS Another option is to include a second, concomitant, summative evaluation in the budget for someone other than the original contractor. This, of course, mandates budgeting more money for the additional evaluation or splitting the budget, which, in some cases, would result in grave budgetary constraints. Further, this might lead contractors to think that you-the original evaluator-are not competent or confident enough to do the evaluation independently. Once again, the contractor may question your motives, abilities, your perception of their costcontainment efforts, or your sensibilities as a business manager. The contractor may simply decide to replace you because of all the professional problems evaluators appear to have.

META-EVALUATIONS Still another approach to the problem is meta-evaluation, which 'this evaluator believes is essential for independent evaluators.· We can submit our evaluations to a colleague and request criticism for our own growth as well as for external substantiation of our work. Such substantiation can then be provided to the contractor and other audiences-such as competitors-as external evidence that the findings were achieved using acceptable methodology, and that they represent the situation as validly as possible under the contractual agreement. This approach to the problem does require the original evaluator to request that a colleague perform such meta-evaluation chores gratis or that he or she build meta-evaluation into the original budget. This may

83

not be the best way to retain friends who are also competent evaluators and, as such, are busy with their own practices. Further, metaevaluation, even when it's provided gratis, is a tough concept to explain and! or sell to a contracting agency who is not really sure it needs to hire anyone to conduct any evaluation at all! Through collaboration with members of the Independent Consultation TIG, however, meta-evaluation does become a viable option. Thanks to TIG member's gracious consent to review an evaluation on a gratis basis, this process is currently being field tested. Although it is certain that this meta-evaluation will be useful for the primary evaluator, and perhaps the meta-evaluator, it will be interesting to determine how acceptable his or her expert critical opinions are to the corporate sponsor of the original evaluation and to other users of the evaluation. It will also be interesting to note how the concept of metaevaluation is perceived by them.

THE CASE FOR ACCEPTING REPEAT CONTRACTS FOR SUMMATIVE EVALUATION As already mentioned, it may also be the case that this externality concern is unjustified and simply not a salient concern for evaluators who practice independently. After all, we do have to accept enough contracts to pay our overhead on a regular basis. However, I think that we must not only practice good evaluation, we must also instruct contractors in good evaluation practices. The field has changed and our professional competencies have, it is hoped, increased over the years. In fact, there may be some good features of such an ongoing arrangement. The original evaluator is more familiar with program operations, key staff members, corporate decision makers, and so on. Therefore, he or she might be better able to contain evaluation costs and ensure that important decision makers and! or consumer groups are included and salient questions are asked. Different evaluation approaches might be used for each succeeding evaluation, which would help us grow professionally through the exploration of new techniques. In addition, we might discover previously unidentified program strengths or weaknesses. There is another bonus of repetitive contracts for the same contractor. Such contractors are more likely to provide sensitive information or to sponsor potentially

84

"dangerous" approaches, such as Critical Competitor Analyses, after they've had some experience with evaluation and a given set of evaluators, and understand the potential benefits. But even if corporate sponsors understand these issues and agree to let an original contractor conduct a second summative evaluation, how do they respond to competitors who charge that the great results were found by someone who is not only on their payroll but is predisposed toward the program because of earlier evaluations that also found good results? How do evaluators justify, to themselves and their colleagues, their own willingness to conduct repetitive summative evaluations for the same program sponsor? Perhaps no such justification is necessary. This article has briefly explored some aspects of the appropriate management of summative evaluations on an external basis by independent consultants. Are there other, better ways to approach this problem? Is this issue intractably problematic for independent evaluation consultants? Or is this essentially a trivial issue on which no more time or space need be spent? The reader is encouraged to develop his or her own thoughts and to share them with the members of the Independent Consultation TIG. Please send comments to: Tara D. Knott, Chairperson Independent Consultation TIG 4646 Poplar, #305 Memphis, TN 38117 (90 I) 683-3463

Microcomputers in Evaluation TIG News As a result of John Bowers's effort, the Microcomputers in Evaluation TIG had an excellent program at Evaluation '86. Annual meeting activities included: • a paneldiscussion on usingmicrocomputersto solveevaluationproblems; • a featured presentation on what we have learned about usingmicrocomputers to automate evaluation and research offices by Steve Frankel,