- Email: [email protected]
An open letter to Ernest house
Letters Letters arc selectedlor thctr PI!rII11CI/CC 10matertal published 111 E\ aluauon Practiceor because lite) discUH issues or problems 01gcneral uuerest to pmCI/LI11g el I/II/(I/or~ t.euer» penauung 10material publi shed 111 EPilia) cnucizc articles or reports, correct errors;pro' ide support or or offer dillaell/ points 01' IL'I\, clanficauons. or addtttonuhnformauon Depending the nature oj th» discussion, authors 01 aruclcs 1/1 qUL'II/OIlIIIC/) be gil CIt all opportunuy 10 reply' in the same IHUL' Preference will be gu en to letters that do 1101 exceed 500 words Those accepted lor publtcauon tnay be edited and shortened
agreement, on
AN OPEN LETTER TO ERNEST HOUSE
Dear ProfessorHouse: I was pleased you viewed my response to your comments on internal evaluation (EP, November 1987) to be of sufficient pertinence to warrant a reply (EP, August 1988), I decided, on this occasion, to respond in an open-letterformatinasmuch as this issueof impartiality exercised by internal evaluationofficesshould beof keen interestto our colleagues. I heartilyurge them to contribute to this dialogue. Effective treatment of the issue of impartialitydemandsincreasedand constructive involvement of the evaluation communitywith its diversity of experience and opinion. The increased trend toward internal evaluation has predictably raised questions and concerns as to whether its components can sustain essential impartiality in the face of institutional pressureswhich, it ic; suggested,seek to limit or enervate the findings. Questions regarding the impartialityand institutional integrity of internal evaluation processesshould receive the fullest exploration,and it is hoped thisopen letter willserve to motivate thosewith uniqueexperiences to join in this dialogue. The evaluation community, however, as pluralistic and tolerant as I believe it to be, may find stressful your hypothetical suggestion for the reciprocal FBI/feminist evaluation you used as an exampleof partiality (EP, August 1988). In the interestof professional tranquil ity, some thingsare best left alone. However, this does not obviate the search for criteria to judge objectivity in internalevaluation settings. 59
60
I was somewhat disappointed that you lumped the advocacy approach I outlined with the feminist theory of Kirkup (1986) and her total abandonment of objectivity. I would like to reaffirm our attachment to rigorous, objective evaluation standards. Only when conclusions have been reached do we work to ensure their visibility in the organizational decision-making machinery. Advocacy evaluation (Sonnichsen, 1988) is offered not as a panacea for impartial evaluations but rather as one approach toward establishing the independence of internal evaluation. In your article (EP, February 1986),you wrote you were unclear how advocacy evaluation could help solve the impartiality problem. The essence of my argument is that the high visibility which accrues to the advocacy role of evaluators affords significant organizational scrutiny to evaluation findings, thereby reducing the inclination for evaluators to engage in biased evaluations. Because internal evaluation in organizations is a public enterprise, the multiple participants, stakeholders, observers, and decision makers, all with individual opinions, act as checks and balances on the objectivity of the evaluators. Aconspiracy to manipulate data unethically in this environment is impractical ifnot impossible (Clifford & Sherman, 1983). Because I believe we could gamer consensus that no evaluation will ever be completely unbiased, independent, and objective-total objectivity being the stuff of mythology-the issue to address becomes that of identifying those clements which will allow evaluators to approach the goal of total objectivity. Scriven (1976) has written that there are no wholly unbiased evaluations, but arrangements do exist to minimize and balance their biases. Since independence and impartiality arc critical attributes for a successful internal evaluation staff, the challenge for internal evaluators is to balance the pressures of organizational interests with the professionalism required for ethical, objective evaluations. Any impairment of the perception of independence and impartiality can damage the reputation of the evaluators and render their efforts ineffectual. ' Impartiality in an organizational environment should be the sine qua non of the internal evaluator, whether it is the objectivity of the logical positivist or the confirmability (Lincoln & Guba, 1985)of the naturalistic inquirer.The perception of credibility of the evaluators is a major contribution in the pursuit of impartiality. Clearly, credibility will not accrue to an evaluation staff that is perceived as subject to manipulation by program managers or interest groups within the organization's hierarchy
61
In the Aristotelian sense, the most powerful source of credibility flows from ethical behavior which implicitly incorporates the notion of objectivity. Credibility and ethical behavior, how ever, are necessary but insufficient ingredients for an impartial evaluation staff. Fortunately, additional safeguards exist that further minimize the potential for biased evaluations. Several incentives unique to internal organizational evaluation units act as motivators for evaluators to remain impartial during the conduct of their activities: •
•
•
•
•
•
•
Organizational //Ioblla)-Owing to the internal transfer policies in many organizations, it is difficult to predict an individual's workingenvironmentor his or her supervisory hierarchyover an extended career Career mobility, 10 thi<; instance. adds a safeguard,albeit indirectly, by limiting the formation of personalloyaltiesand other allegiancesthat may serve to distort the evaluation process, thus assuring the highestlevel of credibilityand ethical performance Expert rCl';ew-Because most evaluations have competing constituencres, resultsare heavilyscrutinizedby knowledgeable persons in the program who WIll likely detect the slightestdeparture from evenhandedness and will unhesitatingly publicize any perceivedbiases Recognition of this heterogeneity is a powerful motivator for the individual evaluator. fostering strict adherence to evaluation standards that can Withstand even the most stringent review, Organizational sc ruuny-s-The results of the evaluation are reviewed in decision-making forums throughout the organization. Public exposure. debate, and critical cornrnentary on research results include triangulation of pcrspecuves (multrphsm) and act to validate fmding« (Cook, 1987). This heterogeneous mixture of viewpoints tends to validate findings, as Cook suggested, and the evaluator's anticipation of thi<; review process minimizes any tendency toward bias Professional ethics and standards-Evaluators have established their own professional ethicsand evaluation standards toguide them throughtheconduct of independent and objectiveevaluations. Organizauonal ethics-Many organizations pride themselves on reputations for ethical conduct. and this ethos is an integral part of the organrzation al culture Success wrthm such an organization requires an evaluator's conformance to the predominant institutional values of honestyand integrity, Organlzattonal locanon-s-li the evaluation office is located sufficientlyclose to the head of the organization and possesses the necessary power base to influence theorgaruzation, theremay notexist any incentive to marupulate data and/or findings. Operaung pllliosophy-Advocacy evaluators, who arc highly Visible in the organization because of their change-agent mentality, may be less subject to political manipulation in the organization.
62
• Accolllltablluy-Evaluation components in public agenciesare subject to both internal and external audits of their actlviucs and outputs. This audit process serves as a final deterrent to the manipulation of data or advancement of findmgsbased on personal biases.
These eight factors, operating in the internal evaluation environment, function to reinforce ethical behavior, and implicitly as well as explicitly contribute to the maintenance of evaluator impartiality. Your criticism that internal evaluators may view organizational issues from the same perspective as the administrators and your call for internal evaluators to become the organization's conscience raise, it seems to me, philosophical and leadership issues. Critically thinking evaluators raising important fundamental issues that are potentially embarrassing to the organization require the assurance of operating from a secure organizational position. To imbue a sense of mission as change agents, strong organizational commitment to evaluation should be extended and supported by effective leadership, thus affording the evaluation staff a sense of security and protection. Heads of evaluationoffices set the tone and atmosphere for the conduct of evaluation work, and their motivation, direction, and instructions to staff reflect the type of evaluations that are undertaken. Once critical issues are raised and recommendations offered, it becomes imperative for the evaluators to identify with the issues. The evaluation tradition of neutrality can be dysfunctional when core organizational issues are examined and change is recommended. Without advocacy, these issues can become the victims of bureaucratic inertia and wither in the heat of political posturing by affected entities. In sum, the transition from external to internal evaluations reported by you (1986), Love (1983), and Patton (1986) requires a comprehensive, thoughtful examination of the bifurcated role evaluators playas both organization employees and organization critics. This conundrum requires not heroismby evaluatorsbut due diligence in the pursuitof objective, impartial evaluationsand a continued effort by both scholars and practitioners to seek guidelines to assist internal evaluators in their work. I hope you agree that it is important to continue this Sisyphean pursuit if the practice of internal evaluation is to develop, flourish, and become an instrument in the growth of organizations. The iterativeprocess of pushing theboulder of impartiality
63
up the hill may erode some of the misconceptions about internal evaluators and result in the development of a framework that both influences organizational activities and assists in the recognition of evaluations as an integral and useful organizational function. I thank you for initiating and sustaining this important question and I hope colleagues will join us in contributing to the dialogue. Dick Sonnichsen Deputy Assistant Director Office of Program Evaluations and Audits Federal Bureau of Investigation U.S. Department of Justice Washington, DC 20535
REFERENCES Clifford. 0 L., & Sherman, P. (1983) Internal evaluation: Integrating program evaluarion and management In A J Love (Ed ). DCIl?!OplIIg cffcc/II Cinternal evaluation:New directions for program evaluation: San Francisco: Jessey-Bass. Cook. T. 0 (1987). Post positivist critical muluphsm In W. R. Shadrsh, Jr. & C. S Reichardt (Eds ), Evaluationstudies reuew annual, Vol 12. Newbury Park:Sage, House, E. R (1986) Internal evaluation £, aluation Practice, 7(1),63·64. House, E. R. (1988) Evaluatingthe FBI A response to Sonnichsen Evaluation Practice. 9(3), 43·46. Kirkup, G. (1986) The Icminist evaluator, In E. R. llouve (Ed). Newdirecttons III education evaluation (pp. 68·~). Lewes,England, and Philadelphia' Falmer. Lincoln, Y. S. & Guba. E. G. (1985) Naturultsuc IIUIUl')·. BeverlyHIII\ Sage. Love. A J. (1983) Editors notes In A. J. Love(Ed). Developing effecII\(' internalevaluation. Ne.. dtrecuonsfor program evaluation San Francisco: Jessey-Bass Patton, M. Q. (1986). Unltzanon-focused evaluauon (2nd ed ) Beverly HIlls:Sage. Scriven, M (1976) Evaluation biJ\ and it~ control. In G. V Gld.\\ (Ed). £\(1lu(l1101I studies reuc... annual (\obI I, pp. J19·139) Beverly Hills: Sage Sonmchsen, R. C. (1987). An internal evaluator responds to Ernest House's views on internal evaluation. Evaluauon Practice, 8(4). 34·36. Sonrnchsen, R. C. (1988) Advocacy cvalu;tion. A model for internal evaluation offices. Evaluation and Program Planning, 11. 141·148
agreement, on
AN OPEN LETTER TO ERNEST HOUSE
Dear ProfessorHouse: I was pleased you viewed my response to your comments on internal evaluation (EP, November 1987) to be of sufficient pertinence to warrant a reply (EP, August 1988), I decided, on this occasion, to respond in an open-letterformatinasmuch as this issueof impartiality exercised by internal evaluationofficesshould beof keen interestto our colleagues. I heartilyurge them to contribute to this dialogue. Effective treatment of the issue of impartialitydemandsincreasedand constructive involvement of the evaluation communitywith its diversity of experience and opinion. The increased trend toward internal evaluation has predictably raised questions and concerns as to whether its components can sustain essential impartiality in the face of institutional pressureswhich, it ic; suggested,seek to limit or enervate the findings. Questions regarding the impartialityand institutional integrity of internal evaluation processesshould receive the fullest exploration,and it is hoped thisopen letter willserve to motivate thosewith uniqueexperiences to join in this dialogue. The evaluation community, however, as pluralistic and tolerant as I believe it to be, may find stressful your hypothetical suggestion for the reciprocal FBI/feminist evaluation you used as an exampleof partiality (EP, August 1988). In the interestof professional tranquil ity, some thingsare best left alone. However, this does not obviate the search for criteria to judge objectivity in internalevaluation settings. 59
60
I was somewhat disappointed that you lumped the advocacy approach I outlined with the feminist theory of Kirkup (1986) and her total abandonment of objectivity. I would like to reaffirm our attachment to rigorous, objective evaluation standards. Only when conclusions have been reached do we work to ensure their visibility in the organizational decision-making machinery. Advocacy evaluation (Sonnichsen, 1988) is offered not as a panacea for impartial evaluations but rather as one approach toward establishing the independence of internal evaluation. In your article (EP, February 1986),you wrote you were unclear how advocacy evaluation could help solve the impartiality problem. The essence of my argument is that the high visibility which accrues to the advocacy role of evaluators affords significant organizational scrutiny to evaluation findings, thereby reducing the inclination for evaluators to engage in biased evaluations. Because internal evaluation in organizations is a public enterprise, the multiple participants, stakeholders, observers, and decision makers, all with individual opinions, act as checks and balances on the objectivity of the evaluators. Aconspiracy to manipulate data unethically in this environment is impractical ifnot impossible (Clifford & Sherman, 1983). Because I believe we could gamer consensus that no evaluation will ever be completely unbiased, independent, and objective-total objectivity being the stuff of mythology-the issue to address becomes that of identifying those clements which will allow evaluators to approach the goal of total objectivity. Scriven (1976) has written that there are no wholly unbiased evaluations, but arrangements do exist to minimize and balance their biases. Since independence and impartiality arc critical attributes for a successful internal evaluation staff, the challenge for internal evaluators is to balance the pressures of organizational interests with the professionalism required for ethical, objective evaluations. Any impairment of the perception of independence and impartiality can damage the reputation of the evaluators and render their efforts ineffectual. ' Impartiality in an organizational environment should be the sine qua non of the internal evaluator, whether it is the objectivity of the logical positivist or the confirmability (Lincoln & Guba, 1985)of the naturalistic inquirer.The perception of credibility of the evaluators is a major contribution in the pursuit of impartiality. Clearly, credibility will not accrue to an evaluation staff that is perceived as subject to manipulation by program managers or interest groups within the organization's hierarchy
61
In the Aristotelian sense, the most powerful source of credibility flows from ethical behavior which implicitly incorporates the notion of objectivity. Credibility and ethical behavior, how ever, are necessary but insufficient ingredients for an impartial evaluation staff. Fortunately, additional safeguards exist that further minimize the potential for biased evaluations. Several incentives unique to internal organizational evaluation units act as motivators for evaluators to remain impartial during the conduct of their activities: •
•
•
•
•
•
•
Organizational //Ioblla)-Owing to the internal transfer policies in many organizations, it is difficult to predict an individual's workingenvironmentor his or her supervisory hierarchyover an extended career Career mobility, 10 thi<; instance. adds a safeguard,albeit indirectly, by limiting the formation of personalloyaltiesand other allegiancesthat may serve to distort the evaluation process, thus assuring the highestlevel of credibilityand ethical performance Expert rCl';ew-Because most evaluations have competing constituencres, resultsare heavilyscrutinizedby knowledgeable persons in the program who WIll likely detect the slightestdeparture from evenhandedness and will unhesitatingly publicize any perceivedbiases Recognition of this heterogeneity is a powerful motivator for the individual evaluator. fostering strict adherence to evaluation standards that can Withstand even the most stringent review, Organizational sc ruuny-s-The results of the evaluation are reviewed in decision-making forums throughout the organization. Public exposure. debate, and critical cornrnentary on research results include triangulation of pcrspecuves (multrphsm) and act to validate fmding« (Cook, 1987). This heterogeneous mixture of viewpoints tends to validate findings, as Cook suggested, and the evaluator's anticipation of thi<; review process minimizes any tendency toward bias Professional ethics and standards-Evaluators have established their own professional ethicsand evaluation standards toguide them throughtheconduct of independent and objectiveevaluations. Organizauonal ethics-Many organizations pride themselves on reputations for ethical conduct. and this ethos is an integral part of the organrzation al culture Success wrthm such an organization requires an evaluator's conformance to the predominant institutional values of honestyand integrity, Organlzattonal locanon-s-li the evaluation office is located sufficientlyclose to the head of the organization and possesses the necessary power base to influence theorgaruzation, theremay notexist any incentive to marupulate data and/or findings. Operaung pllliosophy-Advocacy evaluators, who arc highly Visible in the organization because of their change-agent mentality, may be less subject to political manipulation in the organization.
62
• Accolllltablluy-Evaluation components in public agenciesare subject to both internal and external audits of their actlviucs and outputs. This audit process serves as a final deterrent to the manipulation of data or advancement of findmgsbased on personal biases.
These eight factors, operating in the internal evaluation environment, function to reinforce ethical behavior, and implicitly as well as explicitly contribute to the maintenance of evaluator impartiality. Your criticism that internal evaluators may view organizational issues from the same perspective as the administrators and your call for internal evaluators to become the organization's conscience raise, it seems to me, philosophical and leadership issues. Critically thinking evaluators raising important fundamental issues that are potentially embarrassing to the organization require the assurance of operating from a secure organizational position. To imbue a sense of mission as change agents, strong organizational commitment to evaluation should be extended and supported by effective leadership, thus affording the evaluation staff a sense of security and protection. Heads of evaluationoffices set the tone and atmosphere for the conduct of evaluation work, and their motivation, direction, and instructions to staff reflect the type of evaluations that are undertaken. Once critical issues are raised and recommendations offered, it becomes imperative for the evaluators to identify with the issues. The evaluation tradition of neutrality can be dysfunctional when core organizational issues are examined and change is recommended. Without advocacy, these issues can become the victims of bureaucratic inertia and wither in the heat of political posturing by affected entities. In sum, the transition from external to internal evaluations reported by you (1986), Love (1983), and Patton (1986) requires a comprehensive, thoughtful examination of the bifurcated role evaluators playas both organization employees and organization critics. This conundrum requires not heroismby evaluatorsbut due diligence in the pursuitof objective, impartial evaluationsand a continued effort by both scholars and practitioners to seek guidelines to assist internal evaluators in their work. I hope you agree that it is important to continue this Sisyphean pursuit if the practice of internal evaluation is to develop, flourish, and become an instrument in the growth of organizations. The iterativeprocess of pushing theboulder of impartiality
63
up the hill may erode some of the misconceptions about internal evaluators and result in the development of a framework that both influences organizational activities and assists in the recognition of evaluations as an integral and useful organizational function. I thank you for initiating and sustaining this important question and I hope colleagues will join us in contributing to the dialogue. Dick Sonnichsen Deputy Assistant Director Office of Program Evaluations and Audits Federal Bureau of Investigation U.S. Department of Justice Washington, DC 20535
REFERENCES Clifford. 0 L., & Sherman, P. (1983) Internal evaluation: Integrating program evaluarion and management In A J Love (Ed ). DCIl?!OplIIg cffcc/II Cinternal evaluation:New directions for program evaluation: San Francisco: Jessey-Bass. Cook. T. 0 (1987). Post positivist critical muluphsm In W. R. Shadrsh, Jr. & C. S Reichardt (Eds ), Evaluationstudies reuew annual, Vol 12. Newbury Park:Sage, House, E. R (1986) Internal evaluation £, aluation Practice, 7(1),63·64. House, E. R. (1988) Evaluatingthe FBI A response to Sonnichsen Evaluation Practice. 9(3), 43·46. Kirkup, G. (1986) The Icminist evaluator, In E. R. llouve (Ed). Newdirecttons III education evaluation (pp. 68·~). Lewes,England, and Philadelphia' Falmer. Lincoln, Y. S. & Guba. E. G. (1985) Naturultsuc IIUIUl')·. BeverlyHIII\ Sage. Love. A J. (1983) Editors notes In A. J. Love(Ed). Developing effecII\(' internalevaluation. Ne.. dtrecuonsfor program evaluation San Francisco: Jessey-Bass Patton, M. Q. (1986). Unltzanon-focused evaluauon (2nd ed ) Beverly HIlls:Sage. Scriven, M (1976) Evaluation biJ\ and it~ control. In G. V Gld.\\ (Ed). £\(1lu(l1101I studies reuc... annual (\obI I, pp. J19·139) Beverly Hills: Sage Sonmchsen, R. C. (1987). An internal evaluator responds to Ernest House's views on internal evaluation. Evaluauon Practice, 8(4). 34·36. Sonrnchsen, R. C. (1988) Advocacy cvalu;tion. A model for internal evaluation offices. Evaluation and Program Planning, 11. 141·148