Reciprocal authorities in communal writing assessment: Constructing textual value within a “New politics of inquiry”

ASSESSING 0 Ablex

WRITING

Publishing

4(2) Corp.

1997

pp. 133-167 All rights

ISSN of reproduction

10752935

in any form

reserved

Reciprocal Authorities in Communal Writing Assessment: Constructing Textual Value within a “New Politics of Inquiry” BOB BROAD

Illinois State University Drawing on established methods of interpretive research in composition, this article provides a two-level analysis of the portfolio program at a large, urban, Midwestern university. At the first level, this study details the three major forms of evaluative authority in the program and the distinct types of expertise and knowledge on which each form of authority is based. Administrators’ evaluative authority is based in institutional power and disciplinary knowledge; teachers’ authority is based in their richly contextualized pedagogical knowledge; and judgments by outside instructors are validated based on their status as informed but “distanced” evaluators. At the second level of analysis, the article documents and theorizes the rhetorical and political dynamics by which the three forms of authority interact. Presenting transcriptions of key verbal exchanges and participants’ later reflections on those exchanges, the author explores and maps the contested borders of authority among outside instructors, teachers, and administrators. Because this exploration and mapping provides a glimpse of a portfolio program in transition from a psychometric to a hermeneutic model of textual politics in writing assessment, what emerges is a portrait of the future of communal writing assessment.

We see more and more universities and colleges shifting responsibility for high-stakes assessment decisions out of the hands of solo writing instructors and into the hands of groups of instructor-evaluators. Thus Direct all correspondence Campus

Box 4240,

to:

Normal,

Bob Brood, Department IL 61790-4240.

of English,

Illinois State University,

133

134

B. Broad

communal writing assessment is an increasingly common part of the job for instructors and administrators of composition at the university level. I define communal writing assessment as two or more judges working to reach a joint decision on the basis of a writing pe$ormunce. When judges agree in their group decision-making, their efforts are likely to proceed in a straightforward and congenial manner. When judges disagree, however, communal writing assessment blossoms into something much more complex intellectually and emotionally, rhetorically and politically. Historically, one model for resolving disagreements in communal assessment has predominated. Developed by Paul Diederich and others at the Educational Testing Service, the psychometric model adopted a positivist-scientific stance toward textual value, referring to the correct judgment as the “true score” and to minority judgments as “discrepant” (Diederich, French, & Carlton, 1961). To address the problem of evaluative disagreement, Diederich and his colleagues developed elaborate procedures for “calibration,” using detailed rubrics and extensive practice sessions to bring judges’ evaluative frameworks into alignment. When the psychometric model of calibration succeeded-that is, when judges agreed-the objectivist aura of the process remained intact. Within a consensual setting, participants could believe that their efforts had resulted in the discovery of a true score. When dissent persisted, however-often despite the best efforts of everyone involved-the aura of scientism broke down, and the process of communal writing assessment emerged as an openly rhetorical and political process involving power and persuasion of various kinds, some more democratic, others more autocratic (see Broad, 1994a; Owen, 1985; Reynolds, 1990). This essay reports on the portfolio assessment program at City Universi$, a large, diverse university in the urban mid-Western United States. Breaking with the dominant tradition of communal writing assessment, faculty in the First-Year English Program at City University openly embraced the rhetorical and political character of communal writing assessment. Instead of imagining they were discovering the value of the texts they read and judged together, these administrators and instructors re-conceived the assessment process as one of collaboratively constructing the value of students’ texts and the corresponding pass/fail decision for each student’s performance. This theoretical shift from discovering to constructing textual value required instructor-evaluators at City University to develop new political dynamics of communal assessment. My inquiry focuses on the moment-to-moment, face-to-face details of those new political dynamics. In Fourth Generation Evaluation (1989), Guba and Lincoln articulate their vision of “a hermeneutic dialectic,” the new rhetorical and political

135

Reciprocal Authorities

configuration they wish to promote in communal Within their new model, they explain:

assessment

settings.

[O]ne of the major tasks of the evaluator is to conduct the evaluation in such a way that each group must confront and deal with the constructions of all the others, a process we shall refer to as a hermeneutic dialectic.. .As each group copes with the constructions posed by others, their own constructions alter by virtue of becoming better informed and more sophisticated. Ideally, responsive evaluation seeks to reach consensus on all claims, concerns, and issues at this point, but that is rarely if ever possible. (P. 41)

Three features of Guba and Lincoln’s proposed model distinguish it sharply from the psychometric tradition of communal writing assessment described above. The hermeneutic dialectic presumes that the best evaluations emerge from a decision-making process characterized by: l

Egalitarianism:

“each group must confront and deal with the construc-

tions of all the others,” l

Transformation:

l

better informed”, and Heterodoxy: “consensus..

“their own constructions alter by virtue of becoming

By contrast, the psychometric l

Hierarchy:

.is rarely if ever possible.”

tradition has been characterized

by:

scorers must confront and deal with the constructions of ses-

sion leaders, but not vice versa, l

l

Stability: the correct evaluative framework is assumed to pre-exist the norming session. Therefore scorers are expected to “calibrate” their evaluative frameworks to the framework constructed in advance by administrators rather than constructing or negotiating it as a group; and Orthodoxy: in the name of inter-rater agreement, enforced consensus is the explicit goal of calibration sessions (also called “norming” or “standardization” sessions).

Comparison of these two evaluative models helps to explain why the shift to a hermeneutic theoretical framework requires deep revision of the politics of communal writing assessment. Along with Guba and Lincoln, within the past five years a host of researchers have explored the implications of new theoretical movements for assessment generally (Delandshere & Petrosky, 1994; Hanson, 1993; Moss, 1994; Moss, 1996) and for writing assessment specifically (Allen, 1995; Broad, 1994a; Hourigan, 1991; Huot, 1996; White, 1994; William-

136

B. Broad

son, 1993). Invoking a range of related terms-“hermeneutic,” “interpre“constructivist,” “post-positivist,” “post“postmodern,” tive,” structuralist,” and “rhetorical” -these writers have grappled with, among other things, how evaluators might draw upon differences among their judgments as a resource rather than a deficit. Catching the spirit of this theoretical shift, Edward M. White observes that “Instead of trying to repress or ignore difference, we have begun to understand that we should treasure it.” (1994 p. 15) We have achieved widespread agreement that we should value differences; the necessary next step is to understand how. The texts cited and quoted above lay a substantial theoretical groundwork for a hermeneutic model of value in writing assessment. What has only begun to appear in the literature is a detailed look at the everyday lived experience of compositionists assessing their students’ writing performances within such a model. We have numerous descriptions-mostly written by program administrators-of how programs are imagined or expected to work, but few “inside looks” by institutional outsiders. An important exception is Michael S. Allen’s “Valuing Differences: Portnet’s First Year” (1995). In that article Allen addresses several issues also important to this article. For instance, Allen’s virtual community, “Portnet,” implemented several of the same innovations in shared evaluation found in City University’s portfolio program: l

l

l

including the richly contextualized judgment of the pedagogical insider (i.e., the teacher) in evaluative decisions; bringing the insider’s judgment into discussion and debate with informed outsiders; and valuing the mutual transformation that results from the exchange between insiders and outsiders.

Allen’s methodology also corresponds with mine: we both undertook systematic study of the discourse among participants in communal evaluation, employing established methods of qualitative and narrative inquiry (see “Notes on research methodology” below). An important difference between Allen’s study and mine is that the community I studied was actual, not virtual. Unlike the members of Portnet, faculty in City University’s First-Year English Program saw one another every day in the hallways of their English Department. They necessarily approached one another’s evaluations of student writing wielding starkly unequal and distinct forms of institutional power. (Some evaluators were others’ employers and professors; some evaluators were others’ employees and graduate students.) The most dramatic difference between Portnet and City University is that the discourse at City University resulted in actual students either passing or failing their introductory composition

137

Reciprocal Authorities

courses. On the basis of instructors’ decisions, those students were either allowed or forbidden to proceed toward their undergraduate degrees at City University. As several participants in my study pointed out, such consequences for real, live students whom they taught and knew charged their discussions and decisions with anxiety, doubt, and ethical import. I value Allen’s article for its sound interpretive methodology, informed handling of theoretical issues, and forward-looking treatment of virtual communities conducting writing assessment. However, in addition to valuing differences in a political vacuum we must urgently study how to work with differences in face-to-face communities marked by skewed power relationships. Even more difficult, we must learn how to reach sound “live” decisions as a community without covering up or extirpating differences. The success of administrators and instructors at City University in these two efforts makes their program particularly important for mapping the future of hermeneutic assessment. Overview of Portfolio Assessment at City University’ To be eligible to earn an undergraduate degree from City University, students were required to complete the First-Year English sequence, consisting of English 1, English 2, and English 3. The “Mission Statement” for the First-Year English Program provides detailed descriptions of each course in the sequence, including the following one-sentence summaries of each course: l

l

l

“English 1 is a course in writing about personal experience.” “In [English 21 students will concentrate on reading and writing ments.” English 3 is a “course on writing about literature.”

argu-

To certify students’ writing proficiency, the First-Year English program once used an impromptu writing exam completed at the end of English 3. Two years before I conducted this study, the program shifted to a portfolio assessment program as an integral part of English 1. My study focused specifically on this relatively new portfolio program and its connections to English 1. In the fall quarter during which I conducted my study, three administrators and fifty instructors taught English 1 and participated in the accompanying portfolio program. Figure 1, “Structure of the Portfolio Program at City University,” represents the institutional framework of these participants’ professional relationships. The three administrators-Emily, Kevin, and Terri-each led a Team of instructors through the multiple

B. Broad

138

J-m

16 TA’s and adjuncts

Figure 1.

B

10 TA’s and adjuncts 3 full-time faculty

Team

21 TA’s

Structure of the Portfolio Program at City University

stages of portfolio assessment: mid-term norming, mid-term trios, endterm norming, and end-term trios. To bring community, coherence, and guidance to the efforts of the sixteen trios, administrators conducted two “norming” sessions, one at midterm and the other at the end of the term (just prior to each round of trio meetings). Prior to norming sessions, all participants were to read four sample texts (single essays at mid-term, portfolios at end-term) and decide either to pass or to fail each of those texts. Following each norming session, instructors met in trios. Like the teams participating in norming, trios assessed individual essays at mid-quarter and portfolios (containing four essays each) at the end of the term. The key difference between decisions in norming and decisions in trios was that norming decisions were “dry runs”; trio decisions were “live.” Pass/fail was the only final decision discussed in norming and trio sessions; letter grades for passing work were left up to individual instructors. For the purposes of my analysis, I have labeled trio members as either “Teachers” or “Outside Instructors.” The Teacher was the trio member who worked directly with the student-writer in question throughout the quarter, teaching composition techniques, tailoring assignments, and responding to that student’s writing. Outside Instructors had not taught the student whose work was currently under scrutiny, but they had taught other students. As fellow English 1 instructors, those “outside” evaluators

Reciprocal Authorities

139

were therefore intimately familiar with the established curriculum and methods for teaching the course. Each member of a trio would play both roles in the course of a trio meeting. When the work of a particular instructor’s student was under discussion, that instructor acted, by definition, as the Teacher and her trio mates acted as Outside Instructors. Who was Teacher and who was Outside Instructor therefore varied from moment to moment according to which student’s writing was currently under discussion.3 Notes on Research Methodology During twelve weeks on-site at City University and three months of follow-up interviews, I employed the three methods of inquiry that Patton (1990) designates as distinctive of qualitative research: observations, interviews, and study of documents. Specifically, I observed, made field notes on, and tape-recorded a total of seven norming sessions and fourteen trio sessions. I also tape-recorded semi-structured interviews with three administrators and twelve instructors in the program-some more than once-for a total of twenty-seven interviews. Finally, I collected such program documents as the mission statement, the instructor’s guide to the portfolio program, and sample essays and portfolios. As the fall quarter proceeded, I transcribed tapes of norming sessions, trio sessions, and interviews. I was thus able to ask interviewees to respond to transcripts of specific previous events, statements, and verbal exchanges. Once my data were gathered, I narrowed my focus to a “core group” of nine participants: the three program administrators and six instructors from two trios, one trio from Team A and one trio from Team C (see Figure 1). To avoid interpretive bias, I selected this core group strictly according to methodological benefits, not according to the relevance of their data to any of the emerging themes of analysis. I drew final themes of analysis solely from data generated by this core group of participants; examples to further illustrate and complicate those themes were occasionally drawn from outside the core group. Following Glaser and Strauss’s (1967) “constant comparative method,” I systematically developed, tested, and confiied the insights emerging from this study through a three-stage process of data analysis: concurrent analysis; comprehensive reading; and close reading, coding, and sorting. The first phase, concurrent analysis, took place during the twelve weeks of on-site field work. This phase was marked by the constant interplay of collecting and analyzing data that Mishler (1990) calls “inquiry-guided research.” Phase two covered the several months after field work was complete, and involved a comprehensive reading of all transcripts

140

B. Broad

(approximately 1200 pages), field notes, and documents to generate possible categories of analysis. Guba (1978) terms this the “discovery mode” of data analysis. During this phase ten provisional categories of analysis emerged, evolved, merged, or were eliminated with the goal of representing as fully and truly as possible the experiences of my participants. Moving through the data, I found provisional themes elaborated, complicated, and contradicted, and I added, revised, or eliminated themes accordingly. Subsequently, I moved into what Guba calls “verification mode,” in which I began to focus my data and verify themes. Narrowing the data to approximately 700 pages of transcripts plus documents and field notes, I selected excerpts of transcripts that illustrated and complicated the surviving themes. Grouping these quotations into categories and sub-categories, I finally drafted my written analysis to portray as richly as possible the dynamics I found at work in the program. This final phase of close reading, coding, and sorting yielded two main themes, of which this article presents one: the character and dynamics of three forms of evaluative authority across the portfolio program.4 Having offered a sketch of the structure of relationships and chronology of events in City University’s portfolio program and a brief account of my research methodology, I can now turn to the complex dynamics that evolved as participants negotiated evaluative decisions. Through systematic study of participants’ actions, words, and later reflections, I developed a description and theory of three “reciprocal authorities”: Administrators’, Teachers’, and Outside Instructors’. Each of the three forms of authority is complex; even more intricate was the rhetorical and political interplay among them.

ADMINISTRATORS’ AUTHORITY: KNOWLEDGE AND POWER The three administrators of City University’s First-Year English Program attempted a delicate and sometimes peculiar balancing act. In norming sessions they actively strove to bring coherence and consistency to the program by vigorously promoting evaluative decisions they had previously established among themselves. To accomplish this, the three of them met prior to both rounds of norming sessions to make sure they agreed on the administratively sanctioned evaluative decision for each sample text or portfolio. In these pre-norming administrative trio meetings, they also tried to anticipate possible points of dissent and to generate strategies for countering that dissent. At the same time, administrators limited their surveillance of instructors’ evaluations and protected not only specific divergent judgments but

Reciprocal Authorities

141

also alternative, sometimes conflicting, forms of evaluative authority. Combining appropriate influence over instructors’ evaluative decisions with a variety of limits on that influence, Emily, Kevin, and Terri shaped their complex roles within hermeneutic assessment by adopting a range of administrators’ authoritative postures.

Knowledge Claims and Assertions of Authority The most frequent warrant administrators offered for the preeminence of their judgments was their superior knowledge in three areas: the history and theory of the portfolio program, the entire First-Year English sequence (of which English 1 and its portfolio program were one part), and the field of Rhetoric and Composition. Administrators’ distinctive qualifications as departmental and disciplinary authorities were clearest when contrasted with those of first-year Teaching Assistants. Reflecting on how much “power and responsibility” administrators ought to share with instructors in the program, Kevin (Associate Director of First-Year English) saw the TA’s as distinct from other, more experienced instructors, whether part-time or tenured faculty. While Kevin could envision giving experienced instructors more responsibility and power in the process of articulating and negotiating evaluations of students’ writing, he felt it would be a mistake with the new Teaching Assistants: Yeah, that might be interesting [to give instructors more power] but I also think that it might be a mistake to do that [with] the first year teaching assistants.. .I don’t think that first-year people are in the position to make those kinds of decisions in an informed way.. .I think that beginning TA’s need more direction. I think it’s important for a beginning TA to try to make those decisions but then to have a more experienced voice giving a more experienced reading of the papers.. . That “more experienced voice” might be that of a seasoned instructor or a professor, but more often it would be the voice of an administrator. In another interview, Kevin stated that: there were many times when I genuinely believed that I knew better or I knew more [than my graduate students]. . .there were times when I asserted my authority, and I thought-still think-that that was a legitimate thing to do. So Kevin’s experience and expertise as a teacher, researcher, and administrator gave him both the right and the responsibility to “offer more

142

B. Broad

direction” when it came to defining such evaluative issues as “How should we evaluate these pieces of writing?’ and “How should our program form its evaluative criteria ?” This principle helps to explain why Kevin became agitated and took strong action when a first-year Teaching Assistant directly resisted one of Kevin’s evaluative recommendations. In addition to possessing experience and expertise, administrators had formulated a theoretical vision of the portfolio program (and of the FirstYear English Program) that set them apart from most instructors. In midterm norming for Team C, in the context of arguing to fail a sample text entitled “Anguish,” Kevin stressed the intellectual goals of the First-Year English program and the university: This isn’t a trade school, it’s a university. And part of what we emphasize is the development of self-understanding, self-reflection, critical thinking, the development of communication skills, the ability to communicate--to formulate a message and then to communicate it to someone else in a way that they can understand it. Perhaps in response to speeches such as this, a TA named Laura proposed that Kevin’s authority in norming sessions might be rooted in his possessing “maybe a bigger understanding of what writing should be to the academy.” It was this sense that administrators hold a “broader view” of teaching composition -a sense likely bolstered by Kevin’s occasional references during norming discussions to specific articles and books from “the literature” -that lent administrators’ evaluative decisions a distinctive status. Emily and Kevin also asserted their responsibility and right to “stay the course” of the program’s goals and values over time. Their knowledge of the history of the program-and their central role in shaping and disseminating that history-led them to feel justified in setting their evaluative decisions above those of teachers who may have joined the program only months or even weeks before. Kevin explained why he felt it was “okay” for him to “dominate” a particular discussion during Team C’s mid-term norming: I thought I dominated a lot with the [sample essay] “Gramma Sally” discussion, but I thought also...1 thought my comments were okay there. I was explaining.. .I was explaining the program and our longstanding evaluation standards.

Emily draws in a different way on historical authority. Where Kevin laid claim to knowledge of what was “longstanding” in the program, Emily

Reciprocal Authorities

143

pointed to her own personal, steady, and intense efforts to form and maintain the goals and standards of the program. . . .I certainly do feel very often in running those things that I need to make some points. I need to shape a certain kind of model here.. .I guess, I mean I think it’s coercive, and I think I mean it to be...To some extent, that is, there is a way in which I think that I’ve done a lot of work on this curriculum, I’ve done a lot about this sequence. I wouldn’t want to say that my thinking about that is better than anyone else’s, but I do think that I’m trying to keep a steady eye on what we’re trying to accomplish each step of the way.

for students at

Terri, Assistant to the Director of First-Year English, being a TA herself and not a “real” administrator, made no claims like these for herself. In fact, she frequently pointed out to the instructors with whom she worked how limited her authority was. She did, however, support Kevin’s and Emily’s special status in the program: . . . we assume that the administrators are setting the policy for the program. You know, how did the mission statement get written? It got written because the administrators sat down together and produced it and it supposedly is the voice of the program. It’s the policy, the policies and the theory of the program.

Instructors didn’t always share Terri’s view of how policy for the program should be developed and disseminated. As my analysis of instructors’ authorities will show, instructors sometimes felt they ought to have a more active and more powerful role in shaping evaluative standards for the program-that they should be “the voice of the program,” or at least part of that voice. For administrators, however, their own superior qualifications-knowledge of the theory and history of the program as well as of the field of composition, experience and expertise in teaching, and sheer longevity within the program-legitimized their claims to special status for their evaluative decisions. Terri’s responsibility to “reproduce” (her word) administrative evaluative decisions in the instructors of Team A conflicted with her own and others’ view that she held little of the institutional power and special knowledge that authorized Kevin’s and Emily’s evaluations. In response to this tension, Terri created one of the more innovative and intricate mechanisms of authority found in the program: the ambiguous “we.“5 I observed and recorded Terri’s norming technique three times. During all three of those observations, Terri repeatedly used the first person plural in a manner that intrigued me. Concluding the discussion of most sample texts

144

around which there had been statement of-or a request for evaluation in question. Below have italicized all uses of the

B. Broad

some debate, Terri wrapped up with some confirmation of-“what we think” about the I present several such statements by Terri; I first person plural and key related terms.

It’s not a perfect paper, but I think that this level.. .this level of facility is

what we’re going to be calling passing.

Okay, so do you think that as a group we can say that we can see the sense

of failing this paper at this moment?

. ..if they’re writing this well, they just are squeaking by. That’s what.. .that’s been the consensus of, uh, our thinking about this portfolio.

Okay, I think for your purposes in norming, you can consider that we see these essays in descending order. That in the order they appear here, they are the strongest, less strong, less strong, and the weakest at the bottom. We see them in that order. And that the dividing line between pass and fail is right in the middle. Anybody have problems with that?

This pattern in Terri’s bringing norming discussions to a close left me wondering just who “we” were. In all three of my interviews with Ten-i, we discussed the question. First I asked Terri how she imagined the instructors in Team A interpreted her use of the first-person plural: I would think, if I were sitting out here listening

to somebody say that, “Oh, that person means Emily, and Kevin, and [Teni].” That’s what I would think they mean.. .And I would also think behind that, “Okay, those are the people who define this program theoretically”. . .It’s more than just those people personally. It’s the weight of their common opinion and their position as setting policy for everyone else.

Here and at other moments during interviews, Teni suggested that she said “we” she meant “the administrators of the FYEP.” Another she proposed that “we” meant “the First-Year English program,” turned out that that meant the administrators, too. Other times meant even more:

when time but it “we”

. . .not only Emily and Kevin, but the whole past of the program. The program as I understand it, the mission statement itself, which is the work of many people over many years.. .

145

Reciprocal Authorities

Emily and Kevin; the First-Year English Program; “the whole past of the program.” We can see the possible meanings of “we” proliferating. In fact, Terri was aware of this ambiguity and intentionally used it to negotiate her position as “unauthorized authority.” Bob:

That “we” really seems to have at least more than one meaning. It can mean the people who are in charge, and it can mean all of us [participants in the portfolio program], or it can mean some of us, most of us.

Terri:

Yes. I would prefer for them to interpret it rather than me making clear what I mean by that.

Terri invented a rhetorical strategy by which to tap into an authority to which she didn’t feel she had direct access. By cloaking distinctions between administrators’ evaluative decisions and those of instructors in the norming group, she could fulfill her duty to “provide consistency” to the program even though she lacked Kevin’s and Emily’s knowledge and power. Balance of Power: Self-imposed Limits on Administrative Authority So far we have focused on administrators’ efforts to shape and control instructors’ evaluative decisions. Yet for reasons philosophical, theoretical, personal, and logistical, FYEP administrators also spent a good deal of energy setting limits on their power over “live” evaluative decisions in the program. Even in the midst of making forceful authoritative statements regarding the evaluations of sample texts, for example, administrators nevertheless hinted at the limits of their willingness to wield raw evaluative power. For example, in response to those who wanted to pass “Gramma Sally” Emily commented: I just want to say, you know, on my end of it, ESPECIALLY term, I think this paper absolutely cannot pass.

Although the phrase “on my end of it” catches our attention-and although some tioned how effectively such a qualification tor’s power-that phrase did qualify the Emily’s statement. In an interview, Emily culty in discerning the boundary between

at the mid-

may not be the one that first members of the program quescould attenuate an administramore authoritative aspects of admitted to experiencing diffiwhat was idiosyncratic to her

146

B. Broad

(e.g., she called her very favorable reading of a sample essay entitled “Pops ” “aberrant”) and what applied to the entire program. , . .it’s hard for me here to sort out my own stylistic proclivities, and some sort of absolute, where we ought to, as a program, draw the line.

So Emily struggled to distinguish her personal views from program policy. Leading Team C’s norming sessions, Kevin found himself in a similar predicament. In its meeting to prepare for mid-term norming, the administrative trio had agreed that “Gramma Sally” should fail. Instructors on Team C, however, split exactly evenly (1 l/l 1) in their pass/fail votes on that essay. Wrapping up Team C’s discussion of “Gramma Sally,” Kevin attempted to negotiate the difference between what he “personally” believed and what as an administrator he would authoritatively assert: “the party line.” Because there are real compelling reasons to like this paper. And some people would pass it, you know? I’m just saying I personally think you’re doing the writer more of a favor by not [passling it. I’m not going to be reading all your papers; what you do is your business. Urn, but that’s sort of the party line. Take it for what it’s worth. But there is a fair amount of diversity in the program in terms of what people will pass and what people won’t pass.

Sitting in on this norming session, I wondered about the difference between Kevin’s “just personal” view that “Gramma Sally” should not pass and the corresponding administrative position that, in Emily’s words to Team B, it “absolutely cannot pass.” So I raised the question in my final interview with Kevin. Kevin:

Bob: Kevin: Bob: Kevin: Bob: Kevin:

Well, it’s interesting that the whole sentence is the party line. The party line is, “We have a view on the papers, but what you do in the trios is your business, but this is our view.” And the implication is, “We really hope you agree.” [laughs] Right. But you know, you won’t necessarily. . . .I didn’t know how to understand

the “personally”

part.

That’s my opinion. Okay, so in fact there isn’t a distinction and the party line.

between your opinion

Right, I’m, I guess, merging my own view with the party line.

Reciprocal Authorities

147

Both Emily and Kevin seemed to experience conflicting impulses around the relationship between their own “proclivities’‘-with which instructors could legitimately disagree-and program policy, to which instructors had to submit. At times they wielded a positioned, limited form of authority that not only allowed but welcomed “diversity.” At other times their authority appeared far less open to negotiation or subversion. I explain the ambivalence as characteristic of a program in transition from a psychometric to a hermeneutic model of shared evaluation. Administrators didn’t recoil from very direct assertions of authority when faced with an evaluation they considered “out of bounds.” Away from the fray, however, in quieter moments of reflection, they spoke of norming in terms that suggested their yearning for a different dynamic. Rather than a forum for the exercise of raw power, or even mediated forms of authority like Terri’s “we,” administrators dreamt of norming as an idealized, egalitarian forum free from the conflicts and “coercion” (Emily’s term) that come with asymmetrical distributions of institutional power. They imagined a forum in which an administrator could speak with “just another voice,” more or less persuasive, or could even disappear altogether. Here is one such statement from each of the program’s administrators. Emily:

I wanna say [what I believe] in my voice, whatever weight it carries, which maybe could be just the weight of one person in the program. But I also want to say that I am not prepared to COMPEL anyone to draw the line there.. . .....

Kevin:

I hoped that [teaching assistants] would read the papers carefully and that the ensuing discussion would allow them to articulate for themselves and the rest of the group their own emerging criteria for evaluating writing. That we’d kind of lay that out on the table and get a sense of what people were thinking.. .I also hoped that I could talk with people about my own views of the papers, and try to persuade, which I think I did with varying degrees of success. .....

Ten?

I’d rather be less present there [in norming]. I’d rather be out of that, like more like what I try to do in a classroom, where I kind of try to stay out of it, just set things up for people to speak with each other.

In the context of administrators’ occasional baldly authoritative statements and actions, such egalitarian relations might be difficult to main-

148

B. Broad

tam. In an informal interview, one instructor in the program listed three other aspects of administrators’ relations with instructors that made the “forum” of norming necessarily something other than straightforwardly democratic: 1.

2. 3.

Emily and Kevin determined graduate instructors’ grades and wrote letters of recommendation that affected TA’s opportunities for future graduate education and employment; Emily and Kevin determined all instructors’ immediate employment by deciding who was awarded assistantships; While Terri did not, as we have already noted, hold these institutional powers directly, she reported to Emily and Kevin on the events of her norming sessions. She acted, and was viewed by instructors, as the proxy of the “true” administrators.

At least for this interviewee, administrators’ hopes of rhetorical relations with instructors free of the distorting effects of their own institutional power seemed elusive at best. Not all of the administrators’ efforts to limit their power were confined to idealistic aspirations, however. Program leaders carefully and consciously designed into the workings of the program clear limits on their ability to monitor instructors’ evaluations. While these limits were not absolute, they must be acknowledged as more tangible and effective than those limitations treated above-personalization of administrative opinions or idealization of norming as an egalitarian forum. Trios as a protected space for instructors was a theoretically developed position at which administrators arrived in pursuit of their desire for a “loose system” that had “some play in it.” Emily affirmed the autonomy of trios as she worried about some who met without three members-in other words, trios that weren’t trios: . . .there’s a way in which we’ve intentionally made this sort of a loose system so that urn, so that there is some play in it. On the other hand you know, how loose is too loose?. . .You know, if we have a system of trios but we don’t have trios, ehhh, ya know, how often is that happening and what does that do to the process?

Kevin echoes Emily’s description of the program as “intentionally loose” when he states that the administrators’ “policy is to have as few policies as possible.” Even though Kevin saw fit to call in the TA instructor named Ted after a norming session and “talk with him” in a highly directive way about what was and wasn’t a passing paper in the program, Kevin never intervened in a trio discussion. In fact, Kevin staked out sig-

149

Reciprocal Authorities

nificant autc norming sessic

ny for instructor he explained:

trios. Following

Team C’s mid-term

I don’t want to breathe down people’s necks, and I think even...1 don’t

think I said this today, but I usually say this at portfolio [norming] meetings, but ultimately you’re the one that puts down the grade for the paper. Even if.. .theoretically, even if your portfolio partners fail a student, probably no one’s going to ever find out if you pass that student.. .So I mean it’s uh. ..I don’t want people to DO that, but I mean we’re not checking, you know, so...If you really feel so strongly about it, ultimately it’s your call, you know?

This space for teacher autonomy is based upon more than administrators’ guilt about holding power or a liberal sense of “tolerance.” It rests instead upon their belief that pedagogical integrity and empowerment in the program depend upon instructors’ decisions carrying special weight. In one interview Emily commented, I think that no matter what [administrators] do to shape a program, no matter what materials you develop or whatever, the teacher’s in charge and it’s the teacher’s mind and the teacher’s sensibility and the teacher’s commitment to everything that shapes the class. So we do some things to, you know, have certain kinds of structures, but the class is mainly what the teacher makes it, and it’s appropriate that the judgment also be what the teacher’s judgment is.

While it is difficult finally to reconcile these articulate defenses of teacher autonomy and evaluative diversity with administrators’ most authoritative handling of certain “out of bounds” evaluations, the prevailing culture of the portfolio program at City University did, in fact, protect and promote the special knowledge and authority of instructors in their dual roles as Teachers and Outside Instructors in trios.

INSTRUCTORS’ AUTHORITY: TEACHERS’ SPECIAL KNOWLEDGE AND OUTSIDE INSTRUCTORS’ “COLD READINGS” If norming is the “civilized” territory within which administrators attempt to domesticate instructors’ judgments about students’ writing, trios are the Wild West of communal evaluation at City University. And if asymmetry of institutional power and disciplinary knowledge is the salient fact of instructor-administrator relations, then relative symmetry of power and

B. Broad

150

knowledge among trio-mates is the defining element of trio dynamics. In trios, equality of power/knowledge and absence of administrative control created a forum within which pedagogical and rhetorical sources of authority emerged with greater clarity. The border of authority between Teachers and Outside Instructors was established through the interplay of two non-disciplinary forms of knowledge: “teachers’ special knowledge” and “cold readings” provided by outside evaluators.

“The Real Teacher Knows”: Asserting the Authority of Teachers’ Special Knowledge (TSK) The influence of teachers’ special knowledge was so pervasive and powerful in trio meetings that I assigned it its own acronym. I define TSK as knowledge shared by a Teacher with his or her trio-mates that up until that point only the Teacher knew. From the roughly one hundred transcript excerpts that fit this category, I offer below a sampling of TSK statements, including at least one statement from each of the six instructors in my core group of participants. Grasping the dramatic moment in which TSK was usually presented is key to understanding its impact. Note that each statement was offered by the Teacher of the student whose work was being judged at that moment. It might prove useful for readers of this article to imagine how such statements would affect them if they were Outside Instructors, attempting to formulate a pass/fail judgment of the relevant student’s writing when that student’s Teacher spoke. I know that if he fails he’s going to quit trying. ..... He’s almost paranoid about grammar--that And he’ll come and ask me about stuff.

he’s gonna make a mistake.

..... [She’s] one of my funnier students. She’s a real pain, though.

These two students worked really hard on those, too.

This writer, actually, she’s one of my brightest students.

Yeah, she’s a real good writer. Sweet kid, too.

Reciprocal Authorities

Actually ago.

the surprising

151

thing is she just started to learn English three years

..... Well aside from the fact that he would chew tobacco and spit it out in the can the whole quarter, you know. Just a very juvenile kid. ..... I don’t think he learned a thing the whole quarter.

As these examples suggest, TSK statements included “inside information” from the teacher on a wide range of topics connected with specific students and their writing: writing process, progress as a writer, effort, attitude, life history, classroom behavior, and various kinds of assessments by the teacher-including what the teacher views as the appropriate pass/fail decision for a given text. TSK may have been the single most potent form of authority within City University’s portfolio program. Yet TSK was also contraband. The entire phenomenon of TSK and the special authority that stems from it is an example of what Robert Brooke (1987) called the “under life” of institutional settings, for TSK was officially prohibited from trio proceedings. Administrators, led by Kevin, repeatedly urged trios to make their pass/ fail judgments on the basis of “text only” and to leave context out of the discussion. In other words, administrators wanted trios to decide pass/fail without TSK. Several instructors, too, wanted themselves and their triomates to judge without the influence of the background information teachers could provide. Furthermore, both in the instructors’ guide Portfolios and Teaching English I and in their comments to norming session participants, administrators publicized a clear “‘party line” regarding resolution of trio disputes: in trio decisions, majority rules: If the [classroom] teacher disagrees with an outside reader, another member of the [trio] is called on to break the tie.

You go with the majority view.

Note that these straightforward policies of “text only” and “majority rules” allow Teachers no special status within trios. In fact, these policies closely resemble the objectivist praxis of mainstream communal assessment in which Teachers are excluded on principle. If these policies told

152

B. Broad

the entire story of evaluative authority in trios, one might well ask, ‘Why include the teacher at all?’ Two unofficial policies help to explain not only why Teachers are included in City University’s trios, but also how they wield distinctive authority there. In private, administrators seemed to agree with those instructors who wanted the Teacher’s perspective to carry special weight in trio discussions. While the party line was that the majority rules, in practice, administrators provided two mechanisms by which the teacher could be more than simply one vote among three. The first mechanism was the semi-public “pinch-trio” policy.6 Teachers who were very unhappy being outvoted by trio-mates, could “appeal” to other instructors. If the disgruntled Teacher could find two colleagues to corroborate her judgment, then she could legitimately go with the decision she favored. With a touch of ambivalence, Emily explained the unofficial policy of the pinch-trio: I mean I guess I feel like if the teacher really feels that this [trio’s decision] is way, way out of line and feels that strongly that that teacher should probably run around and get some other readings.. .for instance if I had that student who wrote [the sample essay] “Pops,” and I happened to be in that trio where two people failed it, I think I would take that paper to someone else and say, “You tell me what you think,” and I would see if I could get a lot of other people besides those two people who are crazy. And I would think I had the warrant to say that I wasn’t out of line. But I’m not advocating that on a mass scale.

The second mechanism providing special authority for the teacher I termed the “don’t ask, don’t tell” policy. Although everyone in the program claimed to value the broadened perspective that trio-mates brought to evaluative decisions, most also wanted to protect the Teacher’s special authority. The simple solution to conflict between a Teacher and her trio, then, would be for the Teacher to pass or fail a student as she saw fit regardless of her trio-mates’ judgments. In a private interview even Kevin, the program’s most ardent advocate of Outside Instructors’ “cold readings,” sanctioned the don’t-ask-don’t-tell option: For me the teacher always puts the grade on. So even if the other [trio-] partners fail the paper, the teacher can always pass it. As far as I’m concerned.

Recall also that in a statement quoted earlier from a norming session, Kevin hinted at the don’t-ask-don’t-tell policy when he told instructors,

Reciprocal

Authorities

153

“I’m not going to be reading all your papers; what you do is your business.” Ted, a new TA in Team C, was the most vocal protestor against the majority-rules policy, which he repeatedly denounced as a violation of his academic freedom. In an intriguing twist, Ted is also the member of the program who most forcefully rejects both the pinch-trio and the don’t-askdon’t-tell policies that functioned for others to limit Outside Instructors’ evaluative authority in trios and to bolster the Teachers. Ted observes: I am not interested at all in getting away with breaking the rules. That is of no interest to me. It gets me personally one step farther towards some selfdefined goal, but it only hurts me socially because there is always the possibility that I could be found out and damaged and it hurts me personally because it just damages my own concept of my integrity.

Instructors whose sense of integrity was less closely bound than Ted’s to the official policies of the program seemed to have an easier time accepting the vagaries of communal writing assessment, and felt less apparent need to protest official policies they could easily circumvent by implementing unofficial ones. Negotiating TSK in Trios Both the great shortcoming and the great strength of trios is their juxtaposition of Teachers’ and Outside Instructors’ evaluations. As her trio got underway in its first meeting, Veronica articulated the discomfort she was feeling in a way that illuminates this awkward juxtaposition of perspectives. The first part of Veronica’s complaint implied that her turmoil arose from her luck of TSK in evaluating essays written by students who are strangers to her: It’s really hard, you know, because I’m so used to knowing the person that writes these papers. This is kind of freaking me out because it’s almost like I don’t have any bearings.

This statement suggested that the act of trying to conduct “cold readings” left Veronica feeling “freaked out.” A moment later, however, Veronica revised her initial statement. Reaching back to her prior experience evaluating the (by then defunct) exit exam, she realized that it was not simply her lack of teacher’s knowledge that was “freaking her out.” Rather, it was the juxtaposition of her own lack of knowledge about the writer

154

B. Broad

“behind” the text with the abundance of knowledge about the writer held by the Teacher, her trio-mate, sitting just across the table: But, it’s freaking me out. Really, I didn’t think it would. Maybe because you know these people and I don’t, like when we used to do those exit exam things. At that time I could just make up a person.

Grading the old exit exams, Veronica found evaluating the writing of strangers-in other words, cold readingsdasier because she could ‘Ijust make up a person.” Grouped together in trios, however, Teachers and Outside Instructors had to negotiate and collaboratively construct the writer. This awkward collaboration made trio members’ lives more difficult, their discussions more productive, and their decisions more fair, since their decisions accounted for multiple and divergent perspectives. As Guba and Lincoln predicted, trios found that through a “hermeneutic dialectic” their evaluations became “better informed and more sophisticated.” Whereas Veronica “freaked out” when confronted with the commingling of these conflicting (Teacher vs. Outside Instructor) perspectives on evaluative decisions, administrator Kevin positively valued-and actively sought-that very dissonance: I think that’s one thing I like about the portfolio [program], because you’ve got the teacher’s reading which is colored in part by familiarity with the student-whether you like, or whether you don’t like the student-and then you’ve got the other readings from people who have never met the student: a more distanced reading, and less influenced by other factors. And I think the two evaluations are both useful.

Kevin’s notion that the two readings together are more valuable than either alone points to the genius at the heart of City University’s portfolio program. Refusing to allow either the insider’s or the outsiders’ perspective to dominate, the program instead brings them into dialogue in an egalitarian forum. When we also figure the powerful yet constrained influence of administrators into the program’s dialogical and reciprocal configuration of evaluative authority, we can appreciate the full complexity and power of postmodern communal writing assessment as it was enacted at City University. So the privileges and authority of TSK did not go uncontested in the program. When some instructors concluded that their rights as teachers disallowed any intrusion-whether by administrators or fellow teachersother instructors pressed them hard to join the dialogue and negotiate. Such was the case for the trio made up of Ted, Laura, and Martin.

Reciprocal Authorities

155

“Context” was a word that came up repeatedly when participants talked about why they gave special weight or priority to TSK and therefore to teachers’ evaluations. The notion was that the many hours teachers spend working with students, responding to their writing, and building pedagogical relationships should confer special status on the teacher’s evaluation. Explaining his opposition to the entire project of shared evaluation, Ted, a vocal advocate of complete “teacher autonomy,” put it this way: . . .what disturbs me is the mandate that this [trio] discussion is not in fact a discussion: it determines our [Teachers’] final decisions about our students. And that is very disempowering because I work with my students, I give them comments, I give them suggestions, and I have a relationship with them in part based on the expectations I give them on how they’re going to do at the end of the quarter. And [in the trio] that power is out of my hands.

Because of his relationships with and responsibilities to his students, Ted found intolerable the official policy of “majority rules” in trio decisions. In their first trio meeting, Ted debated at length with his trio-mates Laura and Martin whether he must accept their decision if they as the trio majority disagreed with him as the Teacher. Numerous pages of transcripts from this discussion can be fairly well encapsulated in this bit of conversation in which the trio discovered their disagreement about how to handle disagreements. They foundered specifically on the majority rules policy. Ted:

Wait a minute. If you all think that something shouldn’t pass, and I think it should [pass], then it doesn’t pass?

Laura:

Exactly.

Martin:

Yeah.

Ted:

That’s crap. [laughter]

In their trio discussions with Ted, Martin and Laura patiently, articulately, and firmly stood by the official policy that majority rules in trio decisions. In interviews, however, they recognized the appropriateness of affording the teacher a special form of authority. Martin, for example, acknowledged the teacher’s right to be granted a “stronger part.” I can see [Ted’s] point that, yes, he is the teacher and that he would have more authority . . . even though supposedly you are just one part of the three [in the trio], when it comes to your student, you would be the stronger part.

6. Broad

156

Laura is more restrained than Martin in honoring Ted’s wish for sovereignty over his students’ evaluative fates. She is willing to grant the teacher “ultimate authority,” but only on the strict and prior condition of the teacher’s good faith effort to join the “committee” of the trio: I’ll be honest: in the end if the teacher wants to pass someone, then I don’t...1 think the ultimate, the ultimate authority belongs to the teacher, but the teacher shouldn’t assume that authority until he or she has really talked and really become a part of the committee.

For Laura, dialogue-“really talking”-and community is a pre-requisite for autonomy. Note that this exactly opposes Ted’s formulation, in which autonomy is a firm pre-requisite for dialogue and community. Though Ted was the most vocal defender of instructor autonomy, he was not the only instructor in the trio to dig in his or her heels. Laura, in the face of Ted’s and Martin’s both failing an essay she had passed and passing an essay she had failed, deferred to the trio majority on the first decision, but on the second asserted her authority as teacher unequivocally. Laura:

Well, dards within [first]

you know what? I see why you failed this one. My stanare a little bit different because of how she’s performing the class.. .But I actually could be persuaded that this one failed.

Martin:

Okay.

Laura:

. . .I CANNOT

Martin:

Really? [laughter]

be persuaded

that this [second] one passes.

In a somewhat less confrontational manner, the members of the other trio in my core-group of participants also negotiated at length the interplay of Teachers’ and Outside Instructors’ authorities. Veronica, Sandra, and Rhonda were fully aware of the official policy of “majority rules” in trio meetings, but they were all in agreement that the policy could and should be skirted or modified to take TSK into account. In their mid-term trio meeting, Sandra, Veronica, and Rhonda confirmed with one another that “the party line is that the majority rules.” After confuming with me that “no one will be hearing this” conversation in a way that could identify them, however, Sandra proceeded to complicate the scene a bit, explicitly asserting the primacy of TSK:

1.57

Reciprocal Authorities

Yeah, [majority rules] is the party line. But, I think, like if you two [triomates] were saying like, “Well, you know, you might pass it.. .“, or something-if you weren’t really definite and I was more definite-I think I would go with my own first impression...because I know the person in class and all that stuff. So, in that sense, I think the actual teacher would have the final say.

Both trios in my core group debated and negotiated intensively the border between the two forms of authority found in trios: Teachers’ and Outside Instructors’. While it will prove useful to study trios debating the theory and ethics of TSK in reflective moments, we should also look at TSK in practice in the heated, moment-to-moment decision-making of trio members. After a very lengthy discussion of one student’s portfolio, Laura several times deferred to the authority of Ted, the student’s teacher: . . .the teacher is probably it.

the one who should make the final judgment

on

. ..I guess this is the point where I would gladly put it in the hands of the teacher.

In practice, then, Laura was perhaps more ready to grant Teachers their privileged evaluative authority than in her theorization of trio politics quoted earlier. An exchange between Rhonda and Veronica in the other trio also illustrated the dynamics of Teachers’ vs. Outside Instructors’ interpretations and evaluations of texts. Rhonda doubted the sincerity of the writer/narrator of one essay in a portfolio. Veronica, the writer’s teacher, straightened Rhonda out: Rhonda:

This first thing, about the drug experience,

I didn’t believe him.

Veronica:

He cried, Rhonda.

Rhonda:

Oh, really? Okay. Well, see, that’s not coming through in the paper for me. I didn’t know he wasn’t just doing what he thought you should hear. Now, see, there-the real teacher knows.

“The real teacher knows” could serve nicely as the motto for the concept of TSK as it functioned in the trios I observed. By contrast, “The teacher knows too much” captures the notion behind the third and last source of authority I found in the program: “cold readings” provided by Outside Instructors.

158

B. Broad

Outsiders’ “Cold” Readings as a Check on TSK Cast a cold eye On life, on death. Horseman, pass by! -“Under

Ben Bulben,”

W. B. Yeats

We have already seen limits-policies of text-only and majority-rulesplaced on TSK for the sake of dialogue, community, and the improved validity that comes with including Outside Instructors’ as well as Teachers’ evaluative perspectives. In this final section of data analysis, I show why some in the program claimed that “the teacher knows too much” and that readin s lucking TSK should be sought out, taken seriously, and even privileged. !Z One of the clearest calls for the value of outsiders’ readings comes from John, a member of a trio outside my core group of participants. Along with instructors Dorothy and Leslie, John took part in what I perceived as the most voluble, congenial, even jovial among the trios I observed. Because they had so much to say to one another, their meetings lasted longer than any other trio’s. They also joked and laughed more than any other trio. This congeniality had its benefits: in interviews, all three members of this trio spoke of how much they trusted their trio mates and enjoyed their exchanges with them. John also recognized, however, that some of the TSK-talk undermined what he saw as the chief purpose of the trio: I didn’t like to be given that kind of information, because since I like to be able to have some sort of external check, it sort of ruined the process for me if somebody contaminated that, somebody stepped in and told me what they wanted.

TSK “contaminates” the trio process as John understands it. When one of his trio-mates told him of a student writer “I really liked this girl, she’s made so much improvement,” the process was “sort of ruined.” John was the interviewee who first called my attention to the concept and the positive value of cold readings. In fact, John provided the name for this phenomenon when he said “I tried not to [provide TSK], I tried to give [my students’ texts] cold.” In both John’s estimation and mine as participant-observer, Dorothy was the member of John’s trio who was most strongly inclined to provide TSK along with her students’ texts. In fact, the astonishing frequency, duration, and detail of Dorothy’s TSK speeches may have fueled John’s

Reciprocal Authorities

159

enthusiasm for cold readings. It may seem ironic, then, that Dorothy closely echoes John’s desire for evaluations produced free of the influence of TSK. Yet Dorothy seems conscious of and serious about the advantages of cold readings, and she and John use the same image to illustrate the unique value of Outside Instructors’ readings: they provide a “check.” It was also nice to get this check: am I grading these people accurately, am I giving them a fair grade? Am I being impartial because I like this person, or not impartial because this guy never comes to class on time? Because we’re human, we respond to different people in different ways, and sometimes that affects us more than we know. That was something else I wanted to mention that I liked, was that check for me, that was nice to have.

Cold readings as a check on the Teacher were likewise sought by other participants in my study. Rhonda sought to “guard against” the influences on evaluations of “personal contact” between teacher and student: I think one of the things I was aware of is what I mentioned a few minutes ago, like personal contact with the student and wanting to guard against that...1 mean I think it was something that we were all sort of trying to guard against.

This statement from an interview with Rhonda helps to illuminate the following exchange with her trio-mates: here we see Rhonda working quietly and steadily to question Veronica’s use of TSK. In this case Veronica wanted to fail her student’s essay while Rhonda, the Outside Instructor, voted to pass it: Veronica:

Simply because I’ve seen his first draft and that was nothing.. .So, this is what he worked up for me.

Rhonda:

Some of the other things do seem to be generalities. But I thought he had set it up pretty well, for me at the beginning. So, do you want [Sandra] to read it?

V:

Yeah. [inaudible] I think this is a difficult issue because he’s a really bright kid but he has dialect interference, or whatever you would call that.

R:

See, I didn’t notice that much.

V:

Really?

Rhonda repeatedly countered Veronica’s TSK, providing an “external check.”

160

B. Broad

Shortly following the above exchange in which Rhonda resisted Veronica’s TSK, Veronica wished to provide just such a “check” on behalf of one of Sandra’s students. When Sandra mentioned in advance of Veronica’s reading a particular essay that “To me [Sandra, the Teacher] he’s sort of like a ‘C,’ a low ‘C,’ sort of,” Veronica responded, “I’m sorry, now, you told me.” And a bit later, Veronica found Sandra’s cold reading of an essay by one of Veronica’s students especially helpful. Sandra:

I’ll tell you that he was the only paper I read that I failed.

Veronica:

Immediately?

s:

Yeah.

V:

Okay. That helps me a lot because I’ve been working with him in other ways.

so hard

Within the often agonistic forum of trio meetings, most assertions of the importance of cold readings come from the Outside Instructor. The comment above from Veronica is noteworthy because as the Teacher Veronica’ admitted the limits of her view and felt that Sandra provided a valuable check on her (Veronica’s) reading, a check that “helps me a lot.” I close my analysis of cold readings with observations on a particularly intriguing participant: Martin. Martin’s speech and bearing were marked most of all by diffidence. At moments he was so soft-spoken and noncommittal that his trio-mates Laura and Ted despaired of ever determining his stand on a particular evaluation and proceeded to make decisions without benefit of his “vote.” Other times, however, Martin spoke clearly and repeatedly-though still softly-in defense of his position. In the two episodes that follow, Martin held his ground on the value of his Outsider’s reading in the face of extended and articulate disagreement from his triomates Laura and Ted, two of the most assertive and tenacious speakers in the entire portfolio program. Earlier I highlighted a moment during the mid-term meeting of this trio in which Ted and Martin, the outside readers, passed an essay that Laura, the teacher, had failed. In an attempt to-help Martin and Ted understand her judgment, and presumably to change their minds, Laura let fly with a barrage of TSK, informing her trio-mates about various difficulties with which the student writer struggled. Martin’s resistance to Laura’s speech was succinct and arresting: Laura:

Okay, I want to tell you something about her. She’s been misplaced. She tested into an English lower than mine. I’m not saying that’s what’s causing me even to judge this; she’s amazing, like, if you’re just curious. This woman, you wouldn’t

Reciprocal Authorities

161

believe the experiences she’s had in the past, and she’s not dyslexic, but her family is, like, all identified with different learning disabilities and she has trouble with thoughts in her head, moving to paper, like transferring thoughts to writing. And we even talked about, you know, a tape recorder and stuff like that to help her with her writing. This is like such an ambitious paper and you wouldn’t believe the jump from draft one and draft two, but...Part of me, I guess I felt like [if I passed her] I’d be giving her the wrong message. Martin:

I don’t know. Do you think that’s maybe holding you back? I mean I don’t know.

Martin wondered pointedly whether Laura’s TSK was “holding her back’ from a fair judgment of her student’s work, leading her to judge too harshly so as not to “give the wrong message.” Between two slices of Wonder Bread diffidence (“I don’t know.. .I mean I don’t know”) Martin served Laura a tough slice of advocacy for his and Ted’s “distanced” evaluations-a “cold-reading” cold-cut sandwich. In the face of Ted’s passionate resentment of the majority rules policy guiding trio decisions, Martin also took a quiet, steady stand. Up until this point in the trio session, Martin had maintained a safe silence while Ted and Laura vigorously debated the appropriateness of Outside Instructors’ “incursions” into the Teacher’s evaluative territory. At this moment, however, Laura turned to Martin and directly asked him to state his position. Martin’s response focused on Outside Instructors as representatives of the other instructors in the program, and of the program as a whole: I don’t know. I don’t know if it’s really helping.. .If two other people say it’s failed. The [student] is going to go out to another class [English 2 and English 31 and, I mean, the one that the teacher’s opinion does matter in. I think it matters the most in here. I think if it gets to somebody else and they.. .if the other two people don’t think it’s passing, I think another person is more likely, the next teacher that the student has, would agree, I guess. I’m just saying that it might be better to hold them back...because it’s not going to kill them in ten weeks, if there is going to be a question. Rather than giving them credit and just passing them through. God knows what would happen if they get like a third into the year and they fail every paper they turn in.

Here Martin deployed the now-familiar argument that a trio majority (of two) disagreeing with the Teacher ought to hold sway. This is the standard pro-cold-reading, anti-TSK, majority rules, text-only position, articulated at other moments by administrators and instructors alike. What

B. Broad

162

Martin offered here that was new was more fully contextualized, more fully theorized support for that position. Martin moved the issue beyond a debate about which sort of knowledge-“contextualized” vs. “objective”-is more likely to produce the “right” evaluation. Martin’s was a more probabilistic, more pragmatic, and more democratic argument: that a majority within the trio likely provides a stronger indicator of the views of instructors across the program. Martin is the ideal participant with whose words to close this analysis of the importance of cold readings. Since his overall manner was extremely uncertain and meek, his firmness, even boldness, in asserting the authority of the “check” that outsiders provide helps to highlight the importance of Outside Instructors and their special authority in trios.

CONCLUSION: A NEW POLITICS OF INQUIRY IN COMMUNAL ASSESSMENT The preceding pages have presented a two-level analysis of the First-Year English portfolio program at City University. First, I detailed the three

l

Teacher’s Special Knowledge (TSK)

I

Administrators

Figure 2.

Three Forms of Evaluative Authority

Reciprocal Authorities

163

main forms of authority in the program-Administrators’, Teachers’, and Outside Instructors’ -and the sources of their respective evaluative strengths: power and knowledge, TX, and cold readings (See Figure 2, Three Forms of Evaluative Authority). Second, I collaborated with my participants in exploring and theorizing the intricate dynamics by which those three forms of authority interacted. Presenting transcripts of key verbal exchanges and participants’ later reflections on those exchanges, I mapped the contested borders of authority among the three positions in the program. Figure 3, “Administrators’ Evaluative Authority,” illustrates how norming sessions are the procedure by which administrators wield their considerable evaluative influence in the program; meanwhile, administrators’ idealized portraits of norming and their hands-off approach to trios act as countervailing forces to limit administrators’ power. Figure 4, “Instructors’ Evaluative Authorities,” summarizes the dynamics of evaluative authority in trios. There, two policies or procedures push back from each side of the “contact zone” within which Teachers and Outside Instructors negotiate evaluations. “Text-only” and “majority-rules” policies bolster

Administrators egalitarianportraits

LFGEND l

sources of authority

olicies and procedures as limits on authority

Figure 3.

Administrators’

Evaluative Authorities

164

B. Broad

l

Teacher’s Special Knowledge (TSK)

LEGEND l

sources of authority

as limits on authority

Figure 4.

Instructors’ Evaluative Authorities

the value of Outsiders’ cold readings; at the same time, unofficial policies of the pinch trio and don’t-ask-don’t-tell give Teachers a distinct authoritative edge. We have thus glimpsed the intricate workings of a portfolio program in transition from a psychometric to a hermeneutic model of textual politics. What emerges from all this exploration and mapping is a portrait of the future of communal writing assessment. In “The Psycho-Politics of Error,” James J. Sosnoski traces the implications of poststructuralist theories of textuality for discourse among literary critics. He concludes with this vision of the future: I have argued for the heuristic value of “differences” or heterodoxy in literary criticism.. .To [value differences], however, requires a different notion of error-one that sees difference rather than failure. In short, critics need to develop (I new politics of inquiry, authorizing heterodoxy. An obvious objection can be raised to this valorization of difference: that differences cannot be institutionalized. Though the modem university cannot tolerate differences institutionally, a postmodem one, if allowed to, can. (Sosnoski, 1989, p. 47, emphasis added)

Reciprocal Authorities

165

Throughout this essay, I have drawn liberally on Sosnoski’s phrase “a new politics of inquiry” to characterize what was distinctive about the portfolio program at City University. For the most part, participants in that program shed the theory and politics within which evaluative conflicts have traditionally been handled: the hierarchy, stability, and orthodoxy that so often create a stultifying dynamic of domination and resistance both among instructor-evaluators and between those instructorevaluators and administrators. In place of the old model, City University’s faculty implemented an evaluative authority necessarily and usefully composed of three complementary perspectives. When these multiple authorities disagreed regarding pass/fail decisions, they negotiated and constructed the criteria and standards for evaluation by means of vigorous yet mutually respectful-and mutually empowering-rhetorical exchange. Faculty at City University brought to light a new model for validating assessment decisions, a model in which differences among various authorities are sought out as mutually beneficial rather than avoided as corrupting. In so doing, participants in my study enacted Guba and Lincoln’s Fourth Generation Evaluation and Sosnoski’s new politics of inquiry. Perhaps, according to Sosnoski’s formulation, City University therefore counts as one of the first “postmodern universities.” For my part, I will settle for a more modest claim: by democratizing their politics of evaluation, City University’s program brought vastly enhanced meaning and dignity to the evaluative conflicts that have haunted communal writing assessment to the present day. Acknowledgments: I wish to acknowledge those who made this research possible. Administrators and instructors at City University graciously tolerated my observing, recording, and interviewing antics, and acted as genuine collaborators in my study; George Rundblad and Tom Gerschick, trusty writing-group members, worked over multiple drafts of the manuscript; several colleagues in my department offered useful comments; two reviewers for this journal made suggestions that enabled me to improve the piece significantly; and Carol Hoeniges helped me clarify my thinking about the four figures and then drew them.

NOTES 1. To maintain participants’ anonymity I have fictionalized the name of this university, all personal names (except my own), and any other potentially identifying information (e.g., official titles, place names, and titles of departmental publications). 2. Perhaps the most straightforward way to understand communal portfolio assessment at City University is to know that the program was openly and extensively modeled on Belanoff and Elbow’s program at SUNY-Stonybrook (Belanoff & Elbow, 1991; Elbow

166

B. Broad

& Belanoff, 1991). For further details on the meanings and workings of “trios” and “norming sessions,” readers might look at those published descriptions. 3. I capitalize the names of trio roles so that readers can distinguish between a reference to a Teacher acting a particular role in a trio discussion and a reference to a teacher in the more general sense. 4. Readers seeking a more detailed account of all phases of data gathering and data analysis in this study can read “Chapter 3: Research Methods” in Broad, Working in the City (1994b). The other main theme emerging from this study was the Sisyphean effort of participants in the program to “standardize” their evaluations of students’ writing. For discussion of that theme, see Broad, “Pulling Your Hair Out: Crises of Standardization in Communal Portfolio Assessment.” 5. In my treatment of Terri’s “we” I am indebted to the work of Nedra Reynolds, whose research (1990) on authority and resistance in communal writing assessment alerted me to the richness and complexity of the first-person plural. 6. “Pinch-trio” is my name for the policy; since it had no official status in the program it also had no official name. The same was true for the “don’t-ask-don’t-tell” policy. 7. For an explanation of what makes City University’s portfolio program postmodem, see Guba and Lincoln’s earlier quotation and Sosnoski’s formulation presented in this article’s conclusion. Both suggest that the dialogical character of City University’s program marks a postmodem moment. Alford’s Modern English and the Idea of Language: A Potential Postmodem Practice (1995) names this theoretical and political reciprocity “complementarity” and identifies it as a postmodem turn in both science and rhetoric. My essay (in progress) entitled “Rhetorical Writing Assessment: The Practice and Theory of Complementarity” applies Alford’s work to two recent innovations in writing assessment: multiple and different evaluators (communal writing assessment) and multiple and different artifacts (portfolio assessment). 8. One might notice a close connection between the authority of the informed outsider in trios at City University and the authority of participant-observers in the ethnographic tradition. See, for example, Michael H. Agar’s The Professional Stranger (1980). The participant-observer must steep herself in the culture being studied, but also wields authority as a researcher precisely because she is a stranger to that culture.

REFERENCES Agar, M. H. (1980). The professional stranger. New York Academic Press. Alford, B. E. (1995). Modem English and the idea of language: A potential postmodem pmctice. Unpublished dissertation. Lansing, MI: Michigan State University. Allen, M. S. (1995). Valuing differences: Portnet’s first year. Assessing Writing, Z(l), 6789. Belanoff, P., & Elbow, P. (1991). Using portfolios to increase collaboration and community in a writing program. In P. Belanoff & M. Dickson (Eds.), Port$oZios:Process and product. Portsmouth: BoyntonKook (Heinemann). Broad, B. (Forthcoming). Pulling your hair out: Crises of standardization in communal portfolio assessment. In R. Louth & B. Hain (Eds.), Conflict and consensus: Exploring diversity and standards in the portfolio movement. Urbana, IL: NCTE. Broad, R. L. (1994a). Portfolio scoring: A contradiction in terms. In L. Black, Donald A. Daiker, Jeffrey Sommers, & Gail Stygall (Ed.), New directions in portfolio assessment. Portsmouth: BoyntonKook (Heinemann).

Reciprocal Authorities

167

Broad, R. L. (1994b). Working in the city: Building community and negotiating difference in a po&olio assessment program, Unpublished Dissertation, Miami University, Oxford, OH. Brooke, R. (1987). Underlife and writing instruction. cation, 38(2), 141-53.

College Composition and Communi-

Delandshere, G., & Petrosky, A. R. (1994). Capturing teachers’ knowledge: Performance assessment. Educational Researcher, 23(5), 11-18. Diederich, P. B., French, J. W., & Carlton, S. T. (1961). Factors in the judgment of writing quality. Princeton: Educational Testing Service. Elbow, P., & Belanoff, P. (1991). State University of New York at Stony Brook portfoliobased evaluation program. In P. Belanoff & M. Dickson (Eds.), Portfolios: Process and product (pp. 1- 16). Portsmouth: BoyntonKook (Heinemann). Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine. Guba, E. (1978). Toward a methodology of naturalistic inquiry in educational evaluation. Los Angeles: UCLA Graduate School of Education. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage. Hanson, F. A. (1993). Testing testing: Social consequences of the examined life. Berkeley: University of California Press. Hourigan, M. M. (1991). Poststmctural theory and writing assessment: “Heady, esoteric theory” revisited. Teaching English in the Two-Year College, October, 191-195. Huot, B. (1996). Toward a new theory of writing assessment. College Composition and Communication, 47(4), 549-66. Mishler, E. G. (1990). Validation in inquiry-guided research: The role of exemplars in narrative studies. Harvard Educational Review, 60,415-42. Moss, P. A. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5-12. Moss, P. A. (1996). Enlarging the dialogue in educational measurement: Voices from interpretive research traditions. Educational Researcher, 25(l), 20-28. Owen, D. (1985). None of the above: Behind the myth of scholastic aptitude. Boston: Houghton Mifflin. Patton, M. Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage. Reynolds, N. (1990, October 5-7). Issues of consensus, authority, and resistance: A study/ story of a rater-training session. Paper presented at the University of New Hampshire Conference on the Writing Process. Durham, New Hampshire. Sosnoski, J. J. (1989). The psycho-politics of error. Pre-Text, 10(1-2), 33-52. White, E. M. (1994). Teaching and assessing writing. (2nd ed.). San Francisco: JosseyBass. Williamson, M. M. (1993). An introduction to holistic scoring: The social, historical, and theoretical context for writing assessment. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment. Cresskill, NJ: Hampton. Yeats, W. B. (1962). Selected poems and two plays of William Butler Yeats (M. L. Rosenthal, Trans.). New York: Collier.