- Email: [email protected]
Trustworthiness in evaluation practice: An emphasis on the relational
ARTICLE IN PRESS
Evaluation and Program Planning 30 (2007) 404–409 www.elsevier.com/locate/evalprogplan
Trustworthiness in evaluation practice: An emphasis on the relational$, $$ Sharon F. Rallisa,, Gretchen B. Rossmanb, Rebecca Gajdac a
EPRA, School of Education, University of Massachusetts Amherst, Amherst, MA 01003, USA Center for International Education, University of Massachusetts Amherst, 277 Hills South, 111 Infirmary Way, Amherst, MA 01003, USA c Education Administration Program, University of Massachusetts, 259 Hills House South, 111 Infirmary Way, Amherst, MA 01003, USA
b
Abstract As canons for trustworthiness developed explicitly in the discourse of qualitative inquiry, the emphasis was on procedural matters rather than fundamentally relational ones. A nod was made to the relational in such strategies as ‘‘member checks’’ but the issues of how the evaluator actually relates to participants and to the larger communities of practice and discourse—matters subsumed under moral principles and ethical standards—were often marginalized. This chapter posits that the first consideration in designing and conducting rigorous evaluation inquiry, and in critiquing the results of any research, should be the study’s trustworthiness. Judging a study’s trustworthiness, however, should focus on much more than procedural matters; it should also rely on moral principles and ethical standards which specifically address how we relate to an evaluation’s participants. Central to these considerations are cross-cultural sensibilities about the nuanced meanings associated with any principles and standards. The chapter provides a theoretical framework for this position and offers two illustrative examples in the form of dialogues. r 2007 Elsevier Ltd. All rights reserved. Keywords: Trustworthiness; Ethical evaluation practice; Cross-cultural competence
All research should be grounded in trust. This assertion is a truism but one that remains mysterious in methodological writings. Just what is meant by trust? Trust in what? Trust in whom? And what does the derived term trustworthiness mean? Judgments about the trustworthiness of a study are typically made according to the normative $
This article is based on a performance presented at the annual meeting of the American Evaluation Association in Portleand, OR, in November 2006. Because it was developed as a performance, it does not conform to typical journal article format. There are increasing examples of using performance—both enacted live and written in text—as a medium of research and evaluation dissemination (Alexander, 2005; Rambo, 2007). We would like to thank the two anonymous reviewers of a draft of this article for their helpful feedback. $$ This article is part of the on-going work of Rallis and Rossman; Gajda gracefully agreed to perform the dialogues at the AEA meeting. We would like to thank the two anonymous reviewers of a draft of this article for their helpful feedback. Corresponding author. E-mail addresses: [email protected] (S.F. Rallis), gretchen@ educ.umass.edu (G.B. Rossman), [email protected] (R. Gajda). 0149-7189/$ - see front matter r 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2007.06.006
rules and standards of a particular discipline, often relying on procedural rules as criteria. Traditionally, these rules have stipulated what procedures should be followed to ensure the reliability, validity, objectivity, and generalizability of a study. These cannons imply that the extent to which one can trust the soundness of a study depends on whether appropriate procedures were followed—the technical matters. Was the sample size appropriate? Was the researcher objective? Was the instrumentation reliable? Were the data gathered appropriately? Notably absent are specific discussions of the principles and practice of ethical research, which are often found in discussions separate from the canonical four. When discussed, moreover, the paramount considerations of justice, beneficence, and respect for human beings are often reduced to the procedural matters of gaining informed consent. Given these observations, we argue that the trustworthiness of a study should depend not just on whether the researcher got the technical matters right—whether about instrumentation or the protection of human subjects. Trustworthiness should also be judged by how well the
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
researcher got the relational matters1 right. We argue that these relational matters are central to ethical considerations. Thus, in our text on qualitative research methods, Learning in the Field (Rossman & Rallis, 2003), we define trustworthiness as comprised of both competent practice and ethical considerations for the participants with an underlying demand that the relational matters involved in any research be fore-grounded and privileged. The trustworthiness of a study, then, should be judged not only by how competently it was conducted (according to the norms and standards of a discipline) but also by how ethically it was conducted. For both sets of criteria, however, we argue that reasoning must move beyond the procedural to focus on matters of relationships—with participants, with stakeholders, with peers, with the larger community of discourse. The American Evaluation Association’s Guiding Principles for Evaluators (American Evaluation Association, 2004) recognize the importance of both sets of criteria and both sets of considerations. The principles on Systematic Inquiry (A) and Competence (B) capture the notions of competent practice: for example, evaluators should ‘‘adhere to the highest technical standards appropriate to the methods they use’’ (A.1), and ‘‘practice within the limits of their competence’’ (B.3). The principles on Integrity/Honesty (C) and Respect for People (D) focus on issues subsumed under ethical practice. In these latter two principles, we find discussions of the relational in such language as ‘‘be explicit about their own, their clients’, and other stakeholders’ interests and values related to the evaluation’’ (C.4) and ‘‘carefully judge when the benefits from the evaluation or procedure should be foregone because of potential risks’’ (D.3). Deepening this discussion is principle B.2, which stipulates that ‘‘the members of the evaluation team collectively demonstrate cultural competence’’.2 While these principles provide broad stipulations for evaluation practice, they do not address in depth the relational issues, which are paramount in work with human beings. We argue that, as canons for trustworthiness have developed in the discourse of inquiry, the emphasis has been on procedural matters rather than fundamentally relational ones. Quantitative approaches emphasize procedures to ensure reliability and validity in order to enhance confidence in the findings. Qualitative methodologists introduced the term ‘‘trustworthiness’’ as a substitute for the limited concepts of reliability and validity (as articulated by Lincoln & Guba in their classical 1985 work). Qualitative researchers give a nod to the relational in such 1 We use relational matters to refer to long-term, thoughtful, and sensitive relationships that build as part of sustained research or evaluation work. A full explication of the concept, however, is beyond the scope of this article. 2 Curiously, the deeply relational principle that evaluators be thoughtful about and sensitive to cultural differences is found in the Competence principle (B) rather than the Respect for People principle (D). However, given the typical political wrangling that goes into the crafting of such documents as The Guiding Principles, we are not surprised.
405
strategies as member checks, a decidedly infelicitious term which is matched in technocratic language only by the prescription that the researcher leave ‘‘an audit trail’’ (Lincoln & Guba, 1985). While much has changed in the past 20-plus years, engaging deeply with the demand that respect for the dignity and capacity of all humans should be in the fore-front of inquiry has been superseded by a concern for the procedural. This assertion comes from our years of practice, as academics, evaluators, and supervisors of dissertation research. We have found that, in practice, the issues of how the researcher relates to participants and to the larger communities of practice and discourse have fallen under the jurisdiction of Institutional Review Boards (IRBs). The public discourse—in university hallways and classrooms— focuses on jumping through the IRB hoops rather than thoughtfully and sensitively considering the relational issues that may arise in the conduct of a study. However, in universities and other institutions that receive federal funding, this group is charged with ensuring the protection of human subjects in all research (and much evaluation) conducted under the aegis of that institution. IRBs, thus, serve an important defining and policing role in judging what is considered ethical practice with human subjects. While such boards could take up the educational challenge of discussion and debate about ethical practice, they tend to articulate a set of procedures that simplifies and essentializes what are complex and thorny ethical issues. These and other gatekeeper groups seldom recognize or consider (are unable to or unwilling to?) the multiple possible interpretations of what appear to be straightforward procedures. Informed consent and the protection from harm have dominated procedures governing research activities, becoming emblematic of a codification and bureaucratization of moral reasoning and ethical practice. Informed consent often subsumes the protection from harm; as required by IRBs, however, consent is merely a procedure and does not engage with the larger relational/ethical issues that, we argue, are essential for ethical practice. Notable in the documents provided by IRBs is the lack of ‘‘cultural competence’’ or inter-cultural awareness that much research and evaluation demand. While the AEA Guiding Principles notice cultural competence, they provide no direction for how to mediate between the demands of a US-based (and Euro-centric) university and the sensibilities of cultural groups quite different from the academy. Working in International Development Education, Rossman has engaged in deeply thoughtful discussions about the cultural biases and Euro-centric epistemologic and ontologic assumptions in the bed-rock notions of individual rights (what does this mean in collectivist societies where the individual is blurred and the group is paramount?); free will to agree to participate or withdraw at any time (what does this mean in contexts where participation is not negotiable because of one’s status as a civil servant?); giving one’s consent either by
ARTICLE IN PRESS 406
S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
signature or mark (how is one putting oneself at risk by doing so when living under an authoritarian regime?); protection from harm (how does the researcher know the consequences of participation for village women in a highly patriarchical society?). Sadly, the cultural critique that these questions spring from is not acknowledged in IRB materials. Our work over the past several years (Rallis, 2006a, 2006b; Rallis & Rossman, 2003, 2004; Rallis, Rossman, & Gajda, 2006; Rossman & Rallis, 1998, 2003) has focused on the apparent trivialization of the ethical issues in working with human beings for research or evaluation purposes—matters subsumed under moral principles and ethical standards. We are concerned that procedural matters are increasingly taking precedence over relational ones in institutional discourse and practice. Building relationships with participants has become proceduralized. As we have witnessed over the years, this has had important consequences: graduate students with whom we work speak in coded language about IRB approval, appearing to believe that such approval certifies their research as ‘‘ethical’’. At times, engaging with the thorny issues associated with on-going ethical practice appears tedious, unproductive, and unnecessary. Cultural differences in interpretation of standard forms, such as informed consent, are side-lined; discussions about the ethical representation of what participants have shared are lost; and the epistemological questions about the knowledge claims made in written texts gloss over the crucial relational foundation that generated those claims (see Gunzenhauser, 2006, for a discussion). 1. Dialogues For this paper, we choose to spotlight informed consent because this procedure is central to IRB approval—and because, when viewed merely as a procedure, obtaining informed consent ignores the relationship that must be built between participants and evaluators to result in trustworthy findings. Most students see IRB approval as dependent on their filling in the blanks of a standard informed consent letter that describes the study’s purpose and procedures, invites voluntary (and ensures the right to withdraw) participation, identifies potential risks and benefits, and promises confidentiality. At the seekingIRB-approval stage, they do not often think through and thoughtfully consider what this paper, and the signing of it, might mean to participants. By its very existence, the procedure has taken the place of the deeper reasoning that truly ethical practice demands. Below we offer two dialogues to illustrate the complexities of the concept and practice of obtaining informed consent—and the dilemmas ethical researchers face when building the rich relationships that are essential for trustworthy research or evaluation. Recall that these were originally written as performances; the transposition from an oral and expressive medium into mere text surely limits
their power to evoke. However, the dialogues themselves are real, drawn from our own work. One scenario takes place in a US setting, the other in a small African nation. Authorial commentary and critique are embedded in each dialogue in bold italics. 1.1. Dialogue at the Roofless Women Center Setting the scene: In a large urban city in the US, a group of women who have survived physical abuse have banded together and secured funding to provide counseling and support groups for women in shelters who are escaping abusive relationships. The funding agency wants to know what services have been provided and how they have been used, so has sent an evaluator to interview a sample of women who have received the services and participated in the groups. E ¼ evaluator, P ¼ participant E: I appreciate your taking the time to talk with me. As you know, the Setag Foundation has made possible the Roofless Women Center. They are interested in what your participation has meant to you. When we met last week, you indicated that you might be interested in letting the Foundation know what the Center has meant to you. So today, I’d like to ask you to tell me what you have done with the Roofless Women—and whether you think the counselors and meeting with other women like yourself have helped you. Does that sound ok to you? P: Sure. I’ve thought about it, and it seems like an ok thing to do. The Center has been pretty important to me. But I’ve got a couple of questions. E: Fine. Shoot away. But before we begin, I need to ask you to sign this paper. We call it an informed consent letter—it describes the work for the Foundation and what I’m asking from you. It also says how I’ll use what I learn from our talk. When you sign, it means that you understand the study and that you agree. [Takes out informed consent form]. P: I’m not reading that paper [pushes it away]. E: I’ll go over the paper with you if you’d like—it explains the purpose of the study and promises that whatever you say is confidential. P: What does that mean? E: That no one will hear the tape but me. Your name won’t be on the interview—and then, I’ll erase the tape. That in the report I write, I won’t identify you as the speaker. P: I won’t sign it. Where I come from a signature can get me in trouble. I’m not signing anything. Although the evaluator had begun to build a relationship with the participant, this was clearly not enough for her to feel comfortable. The evaluator has hit smack onto issues of safety and security, with some insensitivity. The participant is appropriately suspicious about signing the document. The possibility that the participant cannot read suggests the
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
additional threat implicit in a mysterious document, and indicates that the evaluator did not think through the context of the women’s lives or whether confronting them with a document might embarrass them. Given the purpose of the Roofless Women’s Center—to create safe spaces for women escaping abusive relationships—to ignore what is most salient to the women—their safety and their life skills—does not respect their human dignity and is clearly not ethical. What else might the evaluator do? E: The paper also tells you there are likely to be no risks, and there may be some benefits. Let me tell you about the benefits. I’m hoping that what we learn can help to improve the services you are receiving. Possibly bring more services for you and others. P: Why should I trust you? I don’t know you. How do I know you won’t change my words? And this report— who will read it? Are they people who can hurt us? And how do I know that my ex-husband might not know about this and then come after me? E: I see what you mean—and I certainly respect your feelings. Well maybe if you sign it, we can just begin talking about your experience with Roofless Women, and you can see how you feel about our conversation. If you aren’t comfortable, you can just stop. The only point of the letter is to ensure that you understand the purpose and are talking to me because you want to.
The participant clearly feels under pressure here. The evaluator is focused on having the participant sign the document rather than on the spirit of the protection of human rights that underlies the notion of ‘informed consent.’ We might question why the evaluator began with this focus on the procedural matter of a signature rather than on the deeper questions of building a relationship with women who have many reasons to mis-trust. P: But why did you choose me? E: Well, the shelter director gave me your name as someone who’s using the services and might benefit from better services. I’d guess you have a story to tell— and you can help me get that story to the funding agency so they can decide if the program is worthwhile. P: Do you mean they might take the services away? E: I’m not saying that, but the Foundation did ask me to give them information on services the program offers. Whatever you can tell me might help them to decide about continued support or maybe to improve the services.
The participant has expressed a quite appropriate question and touched on a difficult issue. From her perspective, if she participates, the program might end. For her, it is a lifeline and a source of a vision of a different way of being in the world. As the evaluator has said, her participation could improve services (but how? she asks) or could lead to the
407
program’s ending. The evaluator then goes on, rather abruptly, to ask: E: Do you want me to tell your story as a Roofless Woman? P: I don’t see how you can tell my story—you have not lived it. And I don’t feel comfortable with signing anything. E: Okay, let’s put the paper aside. What’s most important to me is that I get a chance to hear your experience with the program. I want to get to know you. What would work? P: The story of the roofless women’s shelter is not just my story to tell. You’d need to get to know all of us. E: How can we make that happen? P: Why don’t you come have dinner with us sometime at the shelter? E: I’d like that. When would be a good time? P: Dinner is always at 6:30. Most of us get back to the shelter around 5. Why not come about 6? Tuesday? E: Okay, I’ll be there—and I’ll leave the paper at home. In this dialogue, the participant has taught the evaluator some profound lessons about building relationships. The evaluator was insufficiently sensitive to the experiences of these women—to their life experiences that taught them to mis-trust. The evaluator began with a persistent (and, most likely, annoying, however gently expressed) demand for a signature—the proxy indicator for respect for human dignity—without fully understanding what might be implied in such a demand. Finally, the evaluator moves away from the procedural—the signature—and into the position of learner. The possibilities for building relationships have improved, but the evaluator is fortunate that the participant was willing to open up this door. 1.2. Dialogue in Malawi E ¼ evaluator, P ¼ participant Setting the scene: A small southern African nation that is trying desperately to implement Education for All but has recently enacted policies that prevent students from maximizing their chances to be selected to secondary school. For those not familiar with education systems around the world, selection to secondary school represents a life-changing opportunity. Therefore, policies that indirectly or inadvertently constrain those chances have a profoundly negative effect on the development of the nation’s human resources. The evaluator is using case studies to understand the impact of the policies in selected villages. Because her design calls for interviews of secondary school personnel, she presents the study participants with an informed consent letter. E: I thank you for your willingness to be interviewed. As you know, we want to learn what is happening in your school because of the policies about students’ selection
ARTICLE IN PRESS 408
S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
for secondary schools. Before we get started, I need to have you sign this piece of paper that will give your consent to be interviewed—it is called an informed consent letter. It simply describes the study that I’ve told you about and promises that I’ll protect your privacy. I’ll record your answers, but nobody else will hear the tape, and I won’t use your name in the report. P: You are most welcome. Thank you very much for visiting our school. We are honored. But first I must ask who sent you here? The government? E: No, a local NGO has funded the study—we want to understand the selection process for secondary school: What helps students be selected? What stands in the way of their being selected? How do the policies affect who are your students and how you work with them? P: That is very helpful. Perhaps you should speak with the head teacher first. He will be able to tell you more about this.
The participant is expressing the cultural values of this country in which hierarchy is highly respected. Those in senior positions are considered the ones who have the authority and responsibility to represent the school. This is one meta-message in the dialogue; others are the teacher’s implicit questions, ‘‘Why did you choose me? Have I done anything wrong?’’ E: I have spoken with the head-teacher already. He suggested that I speak with you as leader of Form One [equivalent to ninth grade in the US]. I want to talk with you because you teach the new students and see them daily so you live with the results of the selection process. P: Thank you very much. I would like to talk with you about the problems that stand in the way of our students entering secondary schooling. I am pleased you are interested. I will speak with the head teacher about talking with you and signing the form.
The participant is trying to teach the evaluator about protocol in this context. She is also signaling that she is very reluctant to sign the form. She may well be thinking that she cannot sign the form without explicit approval from the head teacher. E: Ah, but signing the form is essential—I cannot interview unless you sign it. It ensures that you understand the study and that you agree. P: Perhaps the head teacher can give his approval for all the teachers you want to interview.
Here, the participant is signaling her quite-appropriate concerns: that if she signs her name, she will be identifiable, and this could be dangerous. She is also suggesting that, in collectivist societies (and many others, we might add),
everyone in the school will know that she has been interviewed. E: Maybe we should talk more about what I will be doing in the study—and why I am doing the study. I am hoping the results can be helpful to you and to the Government to make sure that more good students can get into secondary school. This is something that all Malawians care about, yes? P: Please do tell me more—I want to understand how the study can help more of our children. And I do want to talk with you. But I cannot sign a paper without talking with my head teacher. Finally, the evaluator begins to understand that a focus on the procedural—the signature—is getting in the way of building a good relationship to learn more about the real challenges facing teachers in secondary schools. The participant has been indirect in her responses, hoping that the evaluator will understand. In this cultural context, however, such indirection is the appropriate way to handle this situation. From a Western perspective, the indirection is difficult to understand. And the focus on the procedural got in the way. 2. Commentary We presented these dialogues as illustrations that informed consent is far more than a procedure; it comprises building a relationship. Obtaining a person’s signature on a letter of consent is at a minimal level a contract. It serves as a proxy indicator of the participant’s trust in the researcher: I trust that you will respect my rights and needs; I trust that you will protect me from harm; I trust that your selection of me as a participant is just. But establishing trust requires more than a contract; it requires interactions grounded in these three moral principles: respect for human persons, beneficence, and justice. But who defines what constitutes respect, beneficence, and justice? Both the evaluator and participants engage in a collaborative process of moral reasoning about how these principles play out in practice. Informed consent as a procedural way to ensure ethical behavior and establish trust has been critiqued in the literature: ‘‘informed consent is a respectful communication process, not a consent form’’ and ‘‘wise researchers’’ modify and even radically alter procedures of informed consent (Sieber 2004, p. 496) in order to establish, develop, and preserve relationships with research participants. Trust-building in evaluation requires moral reasoning that is dialogic, conducted interactively between the evaluator and the participants with the purpose of addressing ambiguities and creating shared understandings. ‘‘To respect another as a moral person is to try to understand his aims and interests from his standpoint and to present him with considerations that enable him to accept the constraints on his conduct’’ (Rawls, 1971, p. 133).
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
Ethical theories, grounded in moral principles, provide direction for evaluators. Consequentialist ethical theories, those focusing on outcomes and results, do not serve to establish trust. Instead, the three principles of respect, beneficence, and justice fall into the category of nonconsequentialist theories, those concerned with the means more than the end. Trustworthiness applies these latter ethical theories. The dialogue using non-consequentialist ethics might consider these questions: What shared values can the evaluator and the participants agree upon? How do both interpret the proposed actions? Specifically, while respect for persons generally means that participants are treated as autonomous agents whose privacy is to be protected, how do these participants define respect? Might their culture value community over individual autonomy? Can these individuals act with autonomy and is their consent absolutely voluntary? Who defines potential harm and benefits? We argue that no form or document can establish this trust. It can articulate the promise but it cannot ensure fulfillment. Only through a relationship between this researcher or evaluator and these specific participants for this particular study can the deep considerations of respect, beneficence, and justice be defined, negotiated, and shared. The relationship that respects human rights and dignity is much more a covenant of mutual agreement than a contract. Definitions of the relationship and what constitutes that respect are reached through moral reasoning with particular emphasis on the ethic of care. And so we move beyond procedures to probity—from technical matters to matters of integrity and moral soundness. References Alexander, B. K. (2005). Performance ethnography: The reenacting and inciting of culture. In N. K. Denzin, & Y. S. Lincoln (Eds.), The Sage
409
handbook of qualitative research (3rd ed., pp. 411–441). Thousand Oaks, CA: Sage. American Evaluation Association. (2004). Guiding principles for evaluators (revised). Retrieved January 16, 2007 from /http://www.eval.org/ Publications/GuidingPrinciples.aspS. Gunzenhauser, M. G. (2006). A moral epistemology of knowing subjects: Theorizing a relational turn for qualitative research. Qualitative Inquiry, 12(3), 621–647. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Rallis, S. F. (2006a). Considering rigor and probity: Qualitative pathways to credible evidence. In Invited paper presented at What Constitutes Credible Evidence in Evaluation and Applied Research? The Claremont Graduate University Stauffer Symposium, Claremont, CA, August. Rallis, S. F. (2006b). A litmus test for scientifically-based research in education: Reasoning with rigor and probity. In Invited plenary at annual meetings of the Northeast Education Research Association, Hudson Valley, NY, October. Rallis, S. F. & Rossman, G. B. (2003). Plenary session: Innovation and Emergence in Qualitative Evaluation. In Invited performance at the annual meeting of the American Evaluation Association, Reno, NV, November. Rallis, S. F., & Rossman, G. B. (2004). Trustworthiness: Beyond procedures to probity. In Paper presented at the annual meeting of the American Educational Research Association, San Diego, April. Rallis, S. F., Rossman, G. B. & Gajda, R. (2006). Trustworthiness: Beyond procedures to probity. In L. Goodyear (Chair), Beyond the guiding principles for evaluators: Professional ethics, cultural norms and trustworthiness in evaluation. Panel at the annual meeting of the American Evaluation Association, Portland, OR, November. Rambo, C. (2007). Handing IRB an unloaded gun. Qualitative Inquiry, 13(3), 353–367. Rawls, J. (1971). A theory of justice. Cambridge, MA: Harvard University Press. Rossman, G. B., & Rallis, S. F. (1998). Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: Sage. Rossman, G. B., & Rallis, S. F. (2003). Learning in the field: An introduction to qualitative research (2nd ed.). Thousand Oaks, CA: Sage. Sieber, J. (2004). Informed consent. The Sage encyclopedia of social science research methods, Vol. 2. Thousand Oaks: Sage Publications (pp. 493–497).
Evaluation and Program Planning 30 (2007) 404–409 www.elsevier.com/locate/evalprogplan
Trustworthiness in evaluation practice: An emphasis on the relational$, $$ Sharon F. Rallisa,, Gretchen B. Rossmanb, Rebecca Gajdac a
EPRA, School of Education, University of Massachusetts Amherst, Amherst, MA 01003, USA Center for International Education, University of Massachusetts Amherst, 277 Hills South, 111 Infirmary Way, Amherst, MA 01003, USA c Education Administration Program, University of Massachusetts, 259 Hills House South, 111 Infirmary Way, Amherst, MA 01003, USA
b
Abstract As canons for trustworthiness developed explicitly in the discourse of qualitative inquiry, the emphasis was on procedural matters rather than fundamentally relational ones. A nod was made to the relational in such strategies as ‘‘member checks’’ but the issues of how the evaluator actually relates to participants and to the larger communities of practice and discourse—matters subsumed under moral principles and ethical standards—were often marginalized. This chapter posits that the first consideration in designing and conducting rigorous evaluation inquiry, and in critiquing the results of any research, should be the study’s trustworthiness. Judging a study’s trustworthiness, however, should focus on much more than procedural matters; it should also rely on moral principles and ethical standards which specifically address how we relate to an evaluation’s participants. Central to these considerations are cross-cultural sensibilities about the nuanced meanings associated with any principles and standards. The chapter provides a theoretical framework for this position and offers two illustrative examples in the form of dialogues. r 2007 Elsevier Ltd. All rights reserved. Keywords: Trustworthiness; Ethical evaluation practice; Cross-cultural competence
All research should be grounded in trust. This assertion is a truism but one that remains mysterious in methodological writings. Just what is meant by trust? Trust in what? Trust in whom? And what does the derived term trustworthiness mean? Judgments about the trustworthiness of a study are typically made according to the normative $
This article is based on a performance presented at the annual meeting of the American Evaluation Association in Portleand, OR, in November 2006. Because it was developed as a performance, it does not conform to typical journal article format. There are increasing examples of using performance—both enacted live and written in text—as a medium of research and evaluation dissemination (Alexander, 2005; Rambo, 2007). We would like to thank the two anonymous reviewers of a draft of this article for their helpful feedback. $$ This article is part of the on-going work of Rallis and Rossman; Gajda gracefully agreed to perform the dialogues at the AEA meeting. We would like to thank the two anonymous reviewers of a draft of this article for their helpful feedback. Corresponding author. E-mail addresses: [email protected] (S.F. Rallis), gretchen@ educ.umass.edu (G.B. Rossman), [email protected] (R. Gajda). 0149-7189/$ - see front matter r 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2007.06.006
rules and standards of a particular discipline, often relying on procedural rules as criteria. Traditionally, these rules have stipulated what procedures should be followed to ensure the reliability, validity, objectivity, and generalizability of a study. These cannons imply that the extent to which one can trust the soundness of a study depends on whether appropriate procedures were followed—the technical matters. Was the sample size appropriate? Was the researcher objective? Was the instrumentation reliable? Were the data gathered appropriately? Notably absent are specific discussions of the principles and practice of ethical research, which are often found in discussions separate from the canonical four. When discussed, moreover, the paramount considerations of justice, beneficence, and respect for human beings are often reduced to the procedural matters of gaining informed consent. Given these observations, we argue that the trustworthiness of a study should depend not just on whether the researcher got the technical matters right—whether about instrumentation or the protection of human subjects. Trustworthiness should also be judged by how well the
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
researcher got the relational matters1 right. We argue that these relational matters are central to ethical considerations. Thus, in our text on qualitative research methods, Learning in the Field (Rossman & Rallis, 2003), we define trustworthiness as comprised of both competent practice and ethical considerations for the participants with an underlying demand that the relational matters involved in any research be fore-grounded and privileged. The trustworthiness of a study, then, should be judged not only by how competently it was conducted (according to the norms and standards of a discipline) but also by how ethically it was conducted. For both sets of criteria, however, we argue that reasoning must move beyond the procedural to focus on matters of relationships—with participants, with stakeholders, with peers, with the larger community of discourse. The American Evaluation Association’s Guiding Principles for Evaluators (American Evaluation Association, 2004) recognize the importance of both sets of criteria and both sets of considerations. The principles on Systematic Inquiry (A) and Competence (B) capture the notions of competent practice: for example, evaluators should ‘‘adhere to the highest technical standards appropriate to the methods they use’’ (A.1), and ‘‘practice within the limits of their competence’’ (B.3). The principles on Integrity/Honesty (C) and Respect for People (D) focus on issues subsumed under ethical practice. In these latter two principles, we find discussions of the relational in such language as ‘‘be explicit about their own, their clients’, and other stakeholders’ interests and values related to the evaluation’’ (C.4) and ‘‘carefully judge when the benefits from the evaluation or procedure should be foregone because of potential risks’’ (D.3). Deepening this discussion is principle B.2, which stipulates that ‘‘the members of the evaluation team collectively demonstrate cultural competence’’.2 While these principles provide broad stipulations for evaluation practice, they do not address in depth the relational issues, which are paramount in work with human beings. We argue that, as canons for trustworthiness have developed in the discourse of inquiry, the emphasis has been on procedural matters rather than fundamentally relational ones. Quantitative approaches emphasize procedures to ensure reliability and validity in order to enhance confidence in the findings. Qualitative methodologists introduced the term ‘‘trustworthiness’’ as a substitute for the limited concepts of reliability and validity (as articulated by Lincoln & Guba in their classical 1985 work). Qualitative researchers give a nod to the relational in such 1 We use relational matters to refer to long-term, thoughtful, and sensitive relationships that build as part of sustained research or evaluation work. A full explication of the concept, however, is beyond the scope of this article. 2 Curiously, the deeply relational principle that evaluators be thoughtful about and sensitive to cultural differences is found in the Competence principle (B) rather than the Respect for People principle (D). However, given the typical political wrangling that goes into the crafting of such documents as The Guiding Principles, we are not surprised.
405
strategies as member checks, a decidedly infelicitious term which is matched in technocratic language only by the prescription that the researcher leave ‘‘an audit trail’’ (Lincoln & Guba, 1985). While much has changed in the past 20-plus years, engaging deeply with the demand that respect for the dignity and capacity of all humans should be in the fore-front of inquiry has been superseded by a concern for the procedural. This assertion comes from our years of practice, as academics, evaluators, and supervisors of dissertation research. We have found that, in practice, the issues of how the researcher relates to participants and to the larger communities of practice and discourse have fallen under the jurisdiction of Institutional Review Boards (IRBs). The public discourse—in university hallways and classrooms— focuses on jumping through the IRB hoops rather than thoughtfully and sensitively considering the relational issues that may arise in the conduct of a study. However, in universities and other institutions that receive federal funding, this group is charged with ensuring the protection of human subjects in all research (and much evaluation) conducted under the aegis of that institution. IRBs, thus, serve an important defining and policing role in judging what is considered ethical practice with human subjects. While such boards could take up the educational challenge of discussion and debate about ethical practice, they tend to articulate a set of procedures that simplifies and essentializes what are complex and thorny ethical issues. These and other gatekeeper groups seldom recognize or consider (are unable to or unwilling to?) the multiple possible interpretations of what appear to be straightforward procedures. Informed consent and the protection from harm have dominated procedures governing research activities, becoming emblematic of a codification and bureaucratization of moral reasoning and ethical practice. Informed consent often subsumes the protection from harm; as required by IRBs, however, consent is merely a procedure and does not engage with the larger relational/ethical issues that, we argue, are essential for ethical practice. Notable in the documents provided by IRBs is the lack of ‘‘cultural competence’’ or inter-cultural awareness that much research and evaluation demand. While the AEA Guiding Principles notice cultural competence, they provide no direction for how to mediate between the demands of a US-based (and Euro-centric) university and the sensibilities of cultural groups quite different from the academy. Working in International Development Education, Rossman has engaged in deeply thoughtful discussions about the cultural biases and Euro-centric epistemologic and ontologic assumptions in the bed-rock notions of individual rights (what does this mean in collectivist societies where the individual is blurred and the group is paramount?); free will to agree to participate or withdraw at any time (what does this mean in contexts where participation is not negotiable because of one’s status as a civil servant?); giving one’s consent either by
ARTICLE IN PRESS 406
S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
signature or mark (how is one putting oneself at risk by doing so when living under an authoritarian regime?); protection from harm (how does the researcher know the consequences of participation for village women in a highly patriarchical society?). Sadly, the cultural critique that these questions spring from is not acknowledged in IRB materials. Our work over the past several years (Rallis, 2006a, 2006b; Rallis & Rossman, 2003, 2004; Rallis, Rossman, & Gajda, 2006; Rossman & Rallis, 1998, 2003) has focused on the apparent trivialization of the ethical issues in working with human beings for research or evaluation purposes—matters subsumed under moral principles and ethical standards. We are concerned that procedural matters are increasingly taking precedence over relational ones in institutional discourse and practice. Building relationships with participants has become proceduralized. As we have witnessed over the years, this has had important consequences: graduate students with whom we work speak in coded language about IRB approval, appearing to believe that such approval certifies their research as ‘‘ethical’’. At times, engaging with the thorny issues associated with on-going ethical practice appears tedious, unproductive, and unnecessary. Cultural differences in interpretation of standard forms, such as informed consent, are side-lined; discussions about the ethical representation of what participants have shared are lost; and the epistemological questions about the knowledge claims made in written texts gloss over the crucial relational foundation that generated those claims (see Gunzenhauser, 2006, for a discussion). 1. Dialogues For this paper, we choose to spotlight informed consent because this procedure is central to IRB approval—and because, when viewed merely as a procedure, obtaining informed consent ignores the relationship that must be built between participants and evaluators to result in trustworthy findings. Most students see IRB approval as dependent on their filling in the blanks of a standard informed consent letter that describes the study’s purpose and procedures, invites voluntary (and ensures the right to withdraw) participation, identifies potential risks and benefits, and promises confidentiality. At the seekingIRB-approval stage, they do not often think through and thoughtfully consider what this paper, and the signing of it, might mean to participants. By its very existence, the procedure has taken the place of the deeper reasoning that truly ethical practice demands. Below we offer two dialogues to illustrate the complexities of the concept and practice of obtaining informed consent—and the dilemmas ethical researchers face when building the rich relationships that are essential for trustworthy research or evaluation. Recall that these were originally written as performances; the transposition from an oral and expressive medium into mere text surely limits
their power to evoke. However, the dialogues themselves are real, drawn from our own work. One scenario takes place in a US setting, the other in a small African nation. Authorial commentary and critique are embedded in each dialogue in bold italics. 1.1. Dialogue at the Roofless Women Center Setting the scene: In a large urban city in the US, a group of women who have survived physical abuse have banded together and secured funding to provide counseling and support groups for women in shelters who are escaping abusive relationships. The funding agency wants to know what services have been provided and how they have been used, so has sent an evaluator to interview a sample of women who have received the services and participated in the groups. E ¼ evaluator, P ¼ participant E: I appreciate your taking the time to talk with me. As you know, the Setag Foundation has made possible the Roofless Women Center. They are interested in what your participation has meant to you. When we met last week, you indicated that you might be interested in letting the Foundation know what the Center has meant to you. So today, I’d like to ask you to tell me what you have done with the Roofless Women—and whether you think the counselors and meeting with other women like yourself have helped you. Does that sound ok to you? P: Sure. I’ve thought about it, and it seems like an ok thing to do. The Center has been pretty important to me. But I’ve got a couple of questions. E: Fine. Shoot away. But before we begin, I need to ask you to sign this paper. We call it an informed consent letter—it describes the work for the Foundation and what I’m asking from you. It also says how I’ll use what I learn from our talk. When you sign, it means that you understand the study and that you agree. [Takes out informed consent form]. P: I’m not reading that paper [pushes it away]. E: I’ll go over the paper with you if you’d like—it explains the purpose of the study and promises that whatever you say is confidential. P: What does that mean? E: That no one will hear the tape but me. Your name won’t be on the interview—and then, I’ll erase the tape. That in the report I write, I won’t identify you as the speaker. P: I won’t sign it. Where I come from a signature can get me in trouble. I’m not signing anything. Although the evaluator had begun to build a relationship with the participant, this was clearly not enough for her to feel comfortable. The evaluator has hit smack onto issues of safety and security, with some insensitivity. The participant is appropriately suspicious about signing the document. The possibility that the participant cannot read suggests the
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
additional threat implicit in a mysterious document, and indicates that the evaluator did not think through the context of the women’s lives or whether confronting them with a document might embarrass them. Given the purpose of the Roofless Women’s Center—to create safe spaces for women escaping abusive relationships—to ignore what is most salient to the women—their safety and their life skills—does not respect their human dignity and is clearly not ethical. What else might the evaluator do? E: The paper also tells you there are likely to be no risks, and there may be some benefits. Let me tell you about the benefits. I’m hoping that what we learn can help to improve the services you are receiving. Possibly bring more services for you and others. P: Why should I trust you? I don’t know you. How do I know you won’t change my words? And this report— who will read it? Are they people who can hurt us? And how do I know that my ex-husband might not know about this and then come after me? E: I see what you mean—and I certainly respect your feelings. Well maybe if you sign it, we can just begin talking about your experience with Roofless Women, and you can see how you feel about our conversation. If you aren’t comfortable, you can just stop. The only point of the letter is to ensure that you understand the purpose and are talking to me because you want to.
The participant clearly feels under pressure here. The evaluator is focused on having the participant sign the document rather than on the spirit of the protection of human rights that underlies the notion of ‘informed consent.’ We might question why the evaluator began with this focus on the procedural matter of a signature rather than on the deeper questions of building a relationship with women who have many reasons to mis-trust. P: But why did you choose me? E: Well, the shelter director gave me your name as someone who’s using the services and might benefit from better services. I’d guess you have a story to tell— and you can help me get that story to the funding agency so they can decide if the program is worthwhile. P: Do you mean they might take the services away? E: I’m not saying that, but the Foundation did ask me to give them information on services the program offers. Whatever you can tell me might help them to decide about continued support or maybe to improve the services.
The participant has expressed a quite appropriate question and touched on a difficult issue. From her perspective, if she participates, the program might end. For her, it is a lifeline and a source of a vision of a different way of being in the world. As the evaluator has said, her participation could improve services (but how? she asks) or could lead to the
407
program’s ending. The evaluator then goes on, rather abruptly, to ask: E: Do you want me to tell your story as a Roofless Woman? P: I don’t see how you can tell my story—you have not lived it. And I don’t feel comfortable with signing anything. E: Okay, let’s put the paper aside. What’s most important to me is that I get a chance to hear your experience with the program. I want to get to know you. What would work? P: The story of the roofless women’s shelter is not just my story to tell. You’d need to get to know all of us. E: How can we make that happen? P: Why don’t you come have dinner with us sometime at the shelter? E: I’d like that. When would be a good time? P: Dinner is always at 6:30. Most of us get back to the shelter around 5. Why not come about 6? Tuesday? E: Okay, I’ll be there—and I’ll leave the paper at home. In this dialogue, the participant has taught the evaluator some profound lessons about building relationships. The evaluator was insufficiently sensitive to the experiences of these women—to their life experiences that taught them to mis-trust. The evaluator began with a persistent (and, most likely, annoying, however gently expressed) demand for a signature—the proxy indicator for respect for human dignity—without fully understanding what might be implied in such a demand. Finally, the evaluator moves away from the procedural—the signature—and into the position of learner. The possibilities for building relationships have improved, but the evaluator is fortunate that the participant was willing to open up this door. 1.2. Dialogue in Malawi E ¼ evaluator, P ¼ participant Setting the scene: A small southern African nation that is trying desperately to implement Education for All but has recently enacted policies that prevent students from maximizing their chances to be selected to secondary school. For those not familiar with education systems around the world, selection to secondary school represents a life-changing opportunity. Therefore, policies that indirectly or inadvertently constrain those chances have a profoundly negative effect on the development of the nation’s human resources. The evaluator is using case studies to understand the impact of the policies in selected villages. Because her design calls for interviews of secondary school personnel, she presents the study participants with an informed consent letter. E: I thank you for your willingness to be interviewed. As you know, we want to learn what is happening in your school because of the policies about students’ selection
ARTICLE IN PRESS 408
S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
for secondary schools. Before we get started, I need to have you sign this piece of paper that will give your consent to be interviewed—it is called an informed consent letter. It simply describes the study that I’ve told you about and promises that I’ll protect your privacy. I’ll record your answers, but nobody else will hear the tape, and I won’t use your name in the report. P: You are most welcome. Thank you very much for visiting our school. We are honored. But first I must ask who sent you here? The government? E: No, a local NGO has funded the study—we want to understand the selection process for secondary school: What helps students be selected? What stands in the way of their being selected? How do the policies affect who are your students and how you work with them? P: That is very helpful. Perhaps you should speak with the head teacher first. He will be able to tell you more about this.
The participant is expressing the cultural values of this country in which hierarchy is highly respected. Those in senior positions are considered the ones who have the authority and responsibility to represent the school. This is one meta-message in the dialogue; others are the teacher’s implicit questions, ‘‘Why did you choose me? Have I done anything wrong?’’ E: I have spoken with the head-teacher already. He suggested that I speak with you as leader of Form One [equivalent to ninth grade in the US]. I want to talk with you because you teach the new students and see them daily so you live with the results of the selection process. P: Thank you very much. I would like to talk with you about the problems that stand in the way of our students entering secondary schooling. I am pleased you are interested. I will speak with the head teacher about talking with you and signing the form.
The participant is trying to teach the evaluator about protocol in this context. She is also signaling that she is very reluctant to sign the form. She may well be thinking that she cannot sign the form without explicit approval from the head teacher. E: Ah, but signing the form is essential—I cannot interview unless you sign it. It ensures that you understand the study and that you agree. P: Perhaps the head teacher can give his approval for all the teachers you want to interview.
Here, the participant is signaling her quite-appropriate concerns: that if she signs her name, she will be identifiable, and this could be dangerous. She is also suggesting that, in collectivist societies (and many others, we might add),
everyone in the school will know that she has been interviewed. E: Maybe we should talk more about what I will be doing in the study—and why I am doing the study. I am hoping the results can be helpful to you and to the Government to make sure that more good students can get into secondary school. This is something that all Malawians care about, yes? P: Please do tell me more—I want to understand how the study can help more of our children. And I do want to talk with you. But I cannot sign a paper without talking with my head teacher. Finally, the evaluator begins to understand that a focus on the procedural—the signature—is getting in the way of building a good relationship to learn more about the real challenges facing teachers in secondary schools. The participant has been indirect in her responses, hoping that the evaluator will understand. In this cultural context, however, such indirection is the appropriate way to handle this situation. From a Western perspective, the indirection is difficult to understand. And the focus on the procedural got in the way. 2. Commentary We presented these dialogues as illustrations that informed consent is far more than a procedure; it comprises building a relationship. Obtaining a person’s signature on a letter of consent is at a minimal level a contract. It serves as a proxy indicator of the participant’s trust in the researcher: I trust that you will respect my rights and needs; I trust that you will protect me from harm; I trust that your selection of me as a participant is just. But establishing trust requires more than a contract; it requires interactions grounded in these three moral principles: respect for human persons, beneficence, and justice. But who defines what constitutes respect, beneficence, and justice? Both the evaluator and participants engage in a collaborative process of moral reasoning about how these principles play out in practice. Informed consent as a procedural way to ensure ethical behavior and establish trust has been critiqued in the literature: ‘‘informed consent is a respectful communication process, not a consent form’’ and ‘‘wise researchers’’ modify and even radically alter procedures of informed consent (Sieber 2004, p. 496) in order to establish, develop, and preserve relationships with research participants. Trust-building in evaluation requires moral reasoning that is dialogic, conducted interactively between the evaluator and the participants with the purpose of addressing ambiguities and creating shared understandings. ‘‘To respect another as a moral person is to try to understand his aims and interests from his standpoint and to present him with considerations that enable him to accept the constraints on his conduct’’ (Rawls, 1971, p. 133).
ARTICLE IN PRESS S.F. Rallis et al. / Evaluation and Program Planning 30 (2007) 404–409
Ethical theories, grounded in moral principles, provide direction for evaluators. Consequentialist ethical theories, those focusing on outcomes and results, do not serve to establish trust. Instead, the three principles of respect, beneficence, and justice fall into the category of nonconsequentialist theories, those concerned with the means more than the end. Trustworthiness applies these latter ethical theories. The dialogue using non-consequentialist ethics might consider these questions: What shared values can the evaluator and the participants agree upon? How do both interpret the proposed actions? Specifically, while respect for persons generally means that participants are treated as autonomous agents whose privacy is to be protected, how do these participants define respect? Might their culture value community over individual autonomy? Can these individuals act with autonomy and is their consent absolutely voluntary? Who defines potential harm and benefits? We argue that no form or document can establish this trust. It can articulate the promise but it cannot ensure fulfillment. Only through a relationship between this researcher or evaluator and these specific participants for this particular study can the deep considerations of respect, beneficence, and justice be defined, negotiated, and shared. The relationship that respects human rights and dignity is much more a covenant of mutual agreement than a contract. Definitions of the relationship and what constitutes that respect are reached through moral reasoning with particular emphasis on the ethic of care. And so we move beyond procedures to probity—from technical matters to matters of integrity and moral soundness. References Alexander, B. K. (2005). Performance ethnography: The reenacting and inciting of culture. In N. K. Denzin, & Y. S. Lincoln (Eds.), The Sage
409
handbook of qualitative research (3rd ed., pp. 411–441). Thousand Oaks, CA: Sage. American Evaluation Association. (2004). Guiding principles for evaluators (revised). Retrieved January 16, 2007 from /http://www.eval.org/ Publications/GuidingPrinciples.aspS. Gunzenhauser, M. G. (2006). A moral epistemology of knowing subjects: Theorizing a relational turn for qualitative research. Qualitative Inquiry, 12(3), 621–647. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Rallis, S. F. (2006a). Considering rigor and probity: Qualitative pathways to credible evidence. In Invited paper presented at What Constitutes Credible Evidence in Evaluation and Applied Research? The Claremont Graduate University Stauffer Symposium, Claremont, CA, August. Rallis, S. F. (2006b). A litmus test for scientifically-based research in education: Reasoning with rigor and probity. In Invited plenary at annual meetings of the Northeast Education Research Association, Hudson Valley, NY, October. Rallis, S. F. & Rossman, G. B. (2003). Plenary session: Innovation and Emergence in Qualitative Evaluation. In Invited performance at the annual meeting of the American Evaluation Association, Reno, NV, November. Rallis, S. F., & Rossman, G. B. (2004). Trustworthiness: Beyond procedures to probity. In Paper presented at the annual meeting of the American Educational Research Association, San Diego, April. Rallis, S. F., Rossman, G. B. & Gajda, R. (2006). Trustworthiness: Beyond procedures to probity. In L. Goodyear (Chair), Beyond the guiding principles for evaluators: Professional ethics, cultural norms and trustworthiness in evaluation. Panel at the annual meeting of the American Evaluation Association, Portland, OR, November. Rambo, C. (2007). Handing IRB an unloaded gun. Qualitative Inquiry, 13(3), 353–367. Rawls, J. (1971). A theory of justice. Cambridge, MA: Harvard University Press. Rossman, G. B., & Rallis, S. F. (1998). Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: Sage. Rossman, G. B., & Rallis, S. F. (2003). Learning in the field: An introduction to qualitative research (2nd ed.). Thousand Oaks, CA: Sage. Sieber, J. (2004). Informed consent. The Sage encyclopedia of social science research methods, Vol. 2. Thousand Oaks: Sage Publications (pp. 493–497).