- Email: [email protected]
Administrators as information brokers
Evaluation and Program Planning, Vol. 10, pp. 315-323, 1987
0149-7189/87 $3.00 + .00 Copyright © 1987 Pergamon Journals Ltd
Printed in the USA. All rights reserved.
ADMINISTRATORS AS INFORMATION BROKERS A Managerial Perspective on Naturalistic Evaluation
CHARLES McCLINTOCK Cornell University
ABSTRACT The managerial perspective on justifying conclusions f r o m naturalistic evaluation is based on how administrators function as brokers who exchange, highlight, and interpret information f o r others. Information brokering is particularly important within a cognitive perspective on organizations, and consists o f process and outcome dimensions. Brokering processes include using evaluative information f o r conceptualizing, motivating action, and monitoring performance. Brokering outcomes refer to conceptual, instrumental, and symbolic results o f information use. The combination o f these dimensions results in a variety o f substantive and methodological criteria that administrators might use to justify knowledge claims f r o m program evaluation.
guide the design, data collection, or analysis of any evaluation, regardless of its epistemological paradigm. This approach to justifying evaluation conclusions is based on a theoretical framework for formative evaluation developed by McClintock (1986). The framework is based on three components: methodology, program theory, and evaluation context that, taken together, provide guidance for the design and analysis of evaluation efforts. The latter two categories are of particular importance for evaluation design and analysis, and consist of substantive issues having to do with the theory that underlies program activities and their expected effects, and the contexts (e.g., community, organizational) that would influence program implementation (see also Bickman, 1987; Scherier, 1987). This framework contrasts with perspectives on evaluation that are solely or predominately methodological (e.g., Cook & Campbell, 1979; Lincoln & Guba, 1985), and whose criteria for justifying conclusions would be epistemological. The approach is similar to Patton's (1978) utilization-focused approach to evaluation, Stufflebeam's (1983) decision-oriented C I P P model, and to Cronbach's (1980, Chapters 2 and 4) position, in particular his discussion of the interests of varied evaluation users in the "policy shaping community." Briefly, the latter perspective highlights evaluation users' (e.g., program
Administrators often serve their organizations as resident ethnographers. Their jobs require them to content analyze communications, have goal-based conversations that are like semi-structured interviews, and sit as participant observers through a constant stream of meetings. The information, inferences, and conclusions drawn from these activities are then conveyed and exchanged among various constituencies inside and outside the organization. It is this process of dealing in and negotiating interpretations of reality that qualifies the administrator as an information broker, and in an informal sense, as a naturalistic evaluator. At the same time administrators often are required to draw conclusions from formal evaluation studies. As users of evaluation data, administrators would justify knowledge claims with reference to the same criteria that they use as informal ethnographers. These criteria are largely determined by administrators' role responsibilities as information brokers. In this paper a conceptual framework will be developed that describes how administrators function as brokers for various types of information as a means of developing, maintaining, or improving programs. From this framework a set of managerial questions and concerns are derived that determine the acceptability of knowledge claims from naturalistic inquiry, and that also could be used to
Requests for reprints should be sent to Charles McClintock, Department of Human Service Studies, College of Human Ecology, Cornell University, Ithaca, NY 14853. 315
316
CHARLES McCLINTOCK
administrators, operating personnel in local sites) distinct information needs and concerns, that in turn are dependent on the type of program and its stage of development. These substantive, as opposed to methodological, concerns largely determine the relevance and validity of evaluation conclusions. Although not the central question in this paper, one might ask how managers would prefer to have evaluatots warrant knowledge claims. The managerial perspective on this question is well stated by Chelimsky THE EPISTEMOLOGICAL
(1987) and Phillips (1987), both of whom cogently argue for the distinctive integrity of scientific method as a process in which knowledge claims are publicly and systematically tested against observation. To the extent that naturalistic inquiry emphasizes relativism and multiple realities, it is less useful to administrators who already are too familiar with that dilemma, and who seek in an evaluator an independent and intentionally (not absolutely) objective party.
POSITION(S) OF MANAGEMENT
Managers are neither scientists nor philosophers, but as practicing information brokers their work bears a resemblance to applied science and philosophy, and thus there are at least implicit epistemological underpinnings for administrative behavior. From the managerial perspective, evaluation conclusions are warranted to the extent that they are consistent with the information brokering requirements of the administrator's role. This pragmatic view would apply regardless of whether or not an evaluation were done within the naturalistic paradigm, and thus it pertains to any systematic empirical product presented to program managers for their use. This position presents a problem philosophically, however, since administrators have to expect to "play the epistemological field" due to the breadth of their information brokering responsibilities. There are three general brokering responsibilities that imply a range of epistemological positions by which managers could justify evaluation conclusions. First, administrators are expected to represent, negotiate, and possibly synthesize the varying perceptions of different program stakeholders. This position emphasizes relativism and the validity of minority as well as majority viewpoints. More importantly, administrators often act like constructionists since they are expected to manage a process in which reality is newly synthesized from the social interplay among various parties to the issue at hand (Gergin, 1985). Second, management often needs data to make short term decisions and adjustments to program activities. Administrators try to step back from the stream of events as i f reality were objective, and to experiment with it in a search for activities that will yield the most desired results. This strategy assumes an epistemology of convergent or fallible realism in which the administrator uses information to obtain a more accurate, though always incomplete or biased, picture of reality that requires continued exploration and revision (Argyris & Sch6n, 1978). Finally, administrators are often called upon to take a position and justify it as if there were no viable alternative version of reality. As leaders with vision or enforcers of the power of a dominant coalition, this managerial requirement results in the administrator as a realist in which
conditions of the world can be known objectively without uncertainty. Paradoxically, it may often be the case that the administrator as realist only uses data symbolically to justify a preference or to signal that a correct or rational decision has been made (Feldman & March, 1981). Under these circumstances evaluation conclusions are warranted when they can be defended as scientifically rational, and when they support a position already endorsed by the administrator or dominant coalition. Despite the range of epistemologies that administrators might use to justify conclusions from evaluation findings, they are most often forced to act like fallible realists, with the emphasis on the latter term. That is, in many situations an administrator must make a choice among several worthy, persuasive, but competing alternatives about what course of action to take. Often that choice will be based on a belief that a particular set of recommendations more accurately reflects reality and will be more likely to achieve an objective. The administrator will often be the first to admit that objectives can shift and that reality can change quickly or be imperfectly and variably assessed, but in the meantime it is often necessary to take a stand. In fact, the popular and research-based literature on management and leadership consistently identifies having a strong vision of reality and a commitment to action as key attributes of the effective administrator (Bennis & Nanus, 1985). The terms fallible or convergent are important too, however, since they are consistent with the prescription that a good administrator tries to listen to different points of view, and indeed almost of definition, is often required to represent the views of all the relevant stakeholders (Whetten, 1984). From the perspective of management there is truth and importance in various versions of organizational life, and the wise administrator tries to acknowledge some validity in all or many of them, recognizing that multiple subjectivity is part of the job. This pragmatic position of management is similar to what Nagel (1986) described as a possibly irreconcilable dualism between objective and subjective dynamics in
Managerial Perspective philosophy. He contended that scientific and technological truth depend on processes that seek to develop knowledge of an objective reality, independent of the particular experiences and perspectives of individual judgement. At the same time, the subjectivity of self is a part of that reality, and knowledge of the world
317
is embodied in individual minds. Justifying conclusions from a managerial perspective requires evaluation methods that strive for an objective (though never finally so) view of the world, that also includes the effects of subjective interpretations of reality.
C O G N I T I O N A N D I N F O R M A T I O N IN O R G A N I Z A T I O N S The idea that administrators, as fallible realists, largely act as brokers of information is particularly significant within the cognitive perspective on organizational behavior (Clark, 1985; Weick, 1979). In this view a critical unit of activity is the sense-making process in which small, often highly segmented groups of individuals try to create sensible and overlapping interpretations of their environment. Sense-making is a process that does not require consensus of interpretation. Dissimilar interpretations can have "equifinality of meaning" when they have compatible implications for action (Donnellon, Gray, & Bougon, 1986). For example, the "Star Wars" defense initiative might be endorsed by conservatives and liberals for very different reasons: the former because they believe it will forestall any arms negotiations with the Soviet Union, and the latter because they believe it can be used as a bargaining chip to achieve massive arms reduction. The importance of sense-making as an organizational process depends on the degree to which ambiguity is characteristic of the work tasks. Ambiguity is the product of the following kinds of conditions: (a) perceptions of problems and solutions are varied and often not explicit, (b) service outcomes are governed more by trial and error than by planned action, (c) the nature of the problems or work is often unclear or changing, (d) goals and causal processes are subject to change and multiple interpretation, and (e) objective standards for evaluation are generally lacking, forcing reliance on socially determined or value-based criteria for judging success (McCaskey, 1982; Weick, 1985). From this perspective, administrators, as well as other professionals, are expected to deal creatively with organizational situations that look like anarchies (March & Olsen, 1976), garbage cans, (Cohen, March, & Olsen, 1972), seesaws (Hedburg, Nystrom, & Starbuck, 1976), and octopoids (Weick, 1977). These metaphors reinforce the importance of information and cognition in organizations, and highlight administrative responsibilities such as sense-making and the interpretation of cause maps, schemas, scripts, symbols, and myths (Sims, Gioia, & Associates, 1986, pp. 1-19). The Environment of H u m a n Service Tasks
The concepts of cognition, ambiguity, and sense-making are especially significant for human service settings in which program evaluation is likely to be prevalent.
Human services refer to those activities whose purpose is to " . . . maintain or improve the general well-being and functioning of people" (Hasenfield & English, 1974, p. 1). While it is possible to classify entire organizations as human services (e.g., schools, hospitals, social service agencies), many types of organizations contain human service activities. For example, administrative, marketing, personnel, planning, training, and internal audit roles, with their emphasis on serving others, can be thought of as embodying human service responsibilities, regardless of the organizational setting (Eden, Jones, & Sims, 1983). Hasenfield and English (1974, pp. 1-32) described several distinctive attributes of human service tasks that highlight the importance of cognitive processes. First, human service professionals work with human beings as their "raw material." Interpersonal perception, moral judgment, social norms and status, and client motivation and autonomy are critical to the effectiveness of service treatments, and to relations between professionals and clients. Second, the goals in human services are often valueladen, ambiguous, and conflicting. For example, approaches to the treatment of juvenile delinquency will be seen by different segments of society as serving such varied purposes as personal growth, reform, control and prevention, or substitution for familial care and responsibility. A given human service encounter may also reflect the differing goals of client, professional, and organization, and a mixture of norms and standards from accrediting, advocacy, professional, or ideological groups. Finally, the "technology" of human service treatments or tasks is often indeterminant, due to variability in human characteristics, motivations, and moods, the clarity of the outcomes that are sought, and the state of theory and knowledge about cause-effect relationships that will produce desired change. To summarize, organizations are thought of as systems for communicating and negotiating the varied, often ambiguous, cognitions and perceptions of reality of groups of stakeholders. Especially in human service settings, interpreting information is a key administrative responsibility. In order to understand better the managerial perspective on drawing conclusions from naturalistic studies, it is necessary to look more closely at how administrators manage information and its interpretation.
318
CHARLES McCLINTOCK INFORMATION
If organizations are viewed correctly as systems of information and cognition, then information can be seen as an important resource to manage. At a macrolevel there is growing recognition that information can be related to improvements in organizational effectiveness, but relatively little is known about the economics and social psychology of information management (Strassman, 1985). Managing information requires attention to several broad areas of administrative responsibility including the following: (a) the acquisition and maintenance of information hardware and software, (b) the design and
MANAGEMENT administration of information systems, (c) staff training, and (d) the utilization of information for program planning, reporting, and management. The emphasis here is on the social psychology o f the utilization process, and in particular on the brokering aspect of utilization; that is, how information is exchanged and interpreted by administrators in order to structure perceptions, motivate action and justify managerial performance. Other important dimensions of utilization include managing forums for information exchange, and the cognitive aspects of causal inference (McClintock, 1987b).
BROKERING INFORMATION USE Administrators are often required to use information in such a way that their programs and decision processes look orderly and rational (Feldman & March, 1981; McGowan, 1976). This situation arises from the expectations of external stakeholders as a prerequisite for accountability, funding, and political support, and of internal stakeholders who often have a need to impose order and rational explanation on events (Weick, 1985). In many instances, however, by virtue of their position in the organization, it is administrators more than anyone else who have a true sense of the fragmented and unplanned course of events, and the variable perceptions of different stakeholders. Thus, information may be used to inform two or more individuals when an opportunity is seen to connect otherwise uncoupled but complementary actions, or to provide new perspectives to groups whose attention and access to information are more narrow than that of the administrator. This description of administrative responsibility, as well as others grounded in behavioral observations (Mintzberg, 1973), conveys an image of the administrator as a broker among multiple, autonomous and often adversarial constituencies inside and outside the organization's boundaries. Information brokering is inherent in the roles and responsibilities of management. Whetten (1984), for example, described several of the information brokering demands on the administrator in higher education. He argued that effective management requires attention to (a) coalition building in which constituent needs and interests are continuously assessed and interrelated, (b) dissemination of new ideas and opportunities, (c) personification of institutional image, (d) interpreting the results of planning and evaluation in a visionary manner, and (e) "over-communicating" by maintaining two-way information flow especially in times of stress or retrenchment. Boyatzis (1985) identified nineteen competencies related to effective managerial performance across 41 different administrative jobs in public and private sector
organizations. Of the 19, the following 10 skills are directly related to the concept o f information brokering; proactivity in information seeking and action, diagnostic use of concepts to find patterns, use of oral presentations, logical thought, conceptualization, social network building, managing group process, perceptual objectivity, concern with close relationships, and ability to convey relevant knowledge. As a broker, the administrator acts as an agent for others by exchanging information among them, alerting them to opportunities, constraints, and interpretations that they might not be aware of, and creating settings and incentives for information to be exchanged, debated, interpreted, and acted on. Administrators may be brokers for many things including cognitions, feelings, power, goals, resources, structural arrangements, and personnel decisions, but the common medium of their work is information. A fundamental responsibility of the human service administrator is to broker information in order to cope with ambiguity, and facilitate understanding and action in the organization. As an information broker, the administrator is faced with the following problems: 1. Information overload and fragmentation. Often there is too much information in too many different forms. The task here is to find ways of processing, relating and conceptualizing information so that it can be absorbed and meaning extracted from it. Organizations have few formal tools, forums, or incentives for coping with information overload and fragmentation (Ackoff, 1967). 2. Relating information to the constraints and social psychological requirements for action. Often information is not timely or in a format that reflects the programmatic structure around which choices and action have to be taken. For example, information in budgets is often not useful for guiding the efforts of personnel due to the prevalence of line item budgeting over program budgeting. By making it difficult to re-
Managerial Perspective late the allocation of equipment, staff or operating funds to desired goals and outcomes, one complicates the perception of others and thus limits the power of the information to focus their attention, commitment, and motivation around programmatic efforts. 3. Having information available on critical indicators of program implementation. Programs may succeed or fail based on the soundness of their underlying theory or because of the ways they are implemented. It is important to establish a monitoring process that allows the organization to assess whether and to what extent a program has been implemented, and to identify those processes that are believed to be critical to its success. These problems establish some of the basic information brokering responsibilities of the administrator, which can be summarized as follows: (a) minimize overload and fragmentation by conceptualizing or framing problems, decisions, and programs in different ways; (b) focus attention on a few decision and action alternatives; (c) monitor the processes of program implementation. In pursuit of these responsibilities, administrators must rely on a variety of informal information sources, such as grapevines, pipelines and myths, and on technical specialists, including evaluators, to monitor information thresholds, and fill information gaps (McClintock, 1985). To summarize, as brokers of information, administrators become spokespersons for the perceptions of others as well as themselves, and they often try to find common ground that will facilitate action. The essence of this process is to use, and govern the use of, language and other symbols, stimulate action from which information about reality will be produced, facilitate interactions among stakeholders in order to share perceptions, and conduct retrospective interpretations of events. The sum of these activities is the management of meaning which is a core responsibility of the administrator's information and communication responsibilities (Bennis & Nanus, 1985, pp. 110-151; Weick, 1985, pp. 132-133).
A Framework for Information Brokering With this general background to information brokering, it is possible to identify a process and an outcome dimension for the concept. The process dimension consists of three distinct but interrelated parts, which can be described as follows: 1. The conceptualizing process, in which large amounts of disparate information are clustered to identify, define, or prioritize problems, processes, opportunities, and past events (Trochim & Linton, 1986). Often the administrator will want to combine perspectives to find new and creative ways of defining reality in order to break deadlocks a m o n g unreconciled perceptions held by others (Sch6n, 1979), or to contrast perspectives that highlight varying assumptions, issues and out-
319
comes (Allison, 1971). The dynamic of conceptualizing is to create, expand or synthesize perceptions of reality. 2. The action process, in which information is used to motivate behavior towards specific decision alternatives, or to create sufficient energy and commitment to implement a decision. The social psychological requirements of concluding a decision or motivating action often mean that reality must be narrowed or simplified, and attention directed to a s!ngle alternative, or to specific standard operating procedures, goals and constraints (Brunsson, 1982). The dynamic of the action process is to narrow the focus of attention to information on one or two alternative choices. 3. The monitoring process, in which one attempts to learn about the effects of chosen decisions, actions or programs. There are two important parts to this activity. First, it is necessary to know whether the p r o g r a m has been implemented and to what extent it is consistent with intended policy. A second process involves making causal inferences about program effects and distinguishing effects expected from the theory that underlies the program from those due to its implementation in a particular agency (Scherier, 1987). The dynamic of the monitoring process is one in which information is used to compare, confirm, or question expectations about human service problems and programs. These are the three components of the process dimension of information brokering. The other dimension of information brokering is defined in terms of the outcomes of information use. As described by Leviton and Hughes (1981), information use can result in the following outcomes: 1. A conceptual outcome where information users gain new understanding of a problem or program. When perceptions are deadlocked, administrators will favor evaluation conclusions that reframe the situation in such a way that perceptual deadlocks are overcome (Donnellon et al., 1986). 2. An instrumental outcome in which information is used to narrow the alternatives, motivate action and achieve short term objectives. 3. A symbolic outcome in which information persuades others about a particular interpretation, or indicates that careful or rational analysis has been undertaken, whether or not it has. Often administrators will favor conclusions that have the strongest scientific character, since these embody the norms of rationality that govern managerial responsibilities. There is some parallelism between these three information brokering outcomes and the three brokering processes presented earlier (i.e., between conceptual outcomes and conceptualizing processes, or instrumental outcomes and action processes). For the present, however, processes and outcomes are conceived as independent dimensions of information brokering. By
C H A R LES McCLINTOCK
320
crosstabulating them in a matrix it is possible to derive a set of questions and concerns that identify specific information brokering responsibilities and requirements, JUSTIFYING CONCLUSIONS Table 1 displays the crosstabulation of brokering processes and outcomes. Illustrative questions and concerns are shown in each cell that define the possible range of an administrator's information brokering responsibilities. Knowledge claims and conclusions would be acceptable to the administrator to the degree that they were consistent with those information brokering responsibilities that were most salient. Based on Table 1 it appears that a broad range of claims and conclusions from naturalistic evaluation could be justified from the managerial perspective. In practice, however, not all cells in the matrix would have equal importance in the context of a particular study. Two metaphors for the role requirements of the administrator can be used to describe how some cells might be weighted more than others. The first metaphor is to think of administrators as high-wire performers (without a safety net below), in that they often are trying to achieve a balance among a set of gravitational forces that are constantly tipping them one direction or another. In terms of Table 1, this image of managerial activity suggests that an administrator will often be balancing the demands of a particular column, row or diagonal. For example, if the program being evaluated were new and controversial
and therefore that would be used by administrators to justify evaluation conclusions.
FROM E V A L U A T I O N
STUDIES
(e.g., making contraceptives available in the schools), the demands of instrumental and symbolic use would be most pressing due to the need for quick action on a politically volatile issue. Management could easily find itself trying to balance the requirements of cells 3, 5, and 9 in the second and third columns of Table 1. Depending on personality characteristics, organizational routines and traditions, and the particular patterns of pressure from others, the administrator could try to balance in the middle and endorse conclusions from cell 5 (the action process), or tilt towards the conclusive value of information in cells 3 or 9 (the conceptual or monitoring processes). In the example of contraceptives in the schools it is likely that the balancing act would be between cells 3 and 5 due to the immediacy of the problem and the intensity of feelings about issues of teenage pregnancy, sexuality and contraception. As noted earlier, the information requirements for action are different than for conceptualizing. To accomplish a decision and motivate action usually means using information to simplify reality in order to minimize cognitive demands and maximize commitment and energy for action. The purpose is to arrive at a single alternative, model or story that stimulates and rewards action. Conceptualizing, on the other hand, re-
TABLE 1 A FRAMEWORK FOR INFORMATION BROKERING IN ORGANIZATIONS Brokering Outcomes Brokering Processes
Conceptualizing
Conceptual
Instrumental
Symbolic
Are perceptual deadlocks broken? Are new concepts, theories, problems, questions or plans discovered?
Is a problem discovered that allows a latent solution to be considered?
Does the information persuade and help gain political or financial support?
Is new thinking generated about the problem or possible solutions?
Does it conform to the values and schema of different stakeholders?
1 Are costs, risks, or benefits redefined or disovered?
Does the information help motivate staff, clients, or others to reach a decision or direct their efforts in a particular direction?
Does the information help justify or explain resource allocation decisions?
Are new behavior patterns, methods of implementation, barriers, or opportunities discovered?
Is the information detailed, timely, and reflective of the complexity and variety of programs as they are being implemented?
Is the information defensible as systematic, trustworthy, or scientific?
Action
Monitoring
3
7
8
9
Managerial Perspective quires information to be taken through many cycles of manipulation and interpretation, and multiple stories are generated in order to acknowledge the perceptions and concerns of as m a n y stakeholders as possible. Weick (1985, p. 130) summarizes this tradeoff between the information demands of action versus conceptualizing as follows: It is relatively hard to gain the best of both worlds. Administrators run the risk of knowing their world well but being unable to do anything about it, or they run the opposite risk of making some decisive change in the world only to discover that they changed the wrong thing and the problem got worse. The second metaphor is to think of the administrator as a frog in a lily pond, hopping to one lily pad and sitting for a while, then to the next, and so on. Sitting is not done, however, to relax and enjoy the sunshine, but rather to satisfy the needs or put out the fires of one stakeholder group while deferring others, eventually trying to get to as many places as possible but in an alternating or sequential fashion. Thus, staying with the contraception example, it might be necessary to: (a) hop to cell 1 in order to find a creative way of thinking of the problem (e.g., define teenage pregnancy as a public health issue, not as a matter of sexuality); (b) then to cell 9 to bring to bear the leverage of systematic, trustworthy, or scientific evidence on the seriousness of the situation; (c) then some quick hops between cell 3, to develop varied portrayals of the program that will calm the fears and attend to the concerns of the school board, PTA, students, religious groups and human service professionals, and cell 4 to check with the lawyers on the legal liability of the school in cases where a student who receives contraceptives subsequently gets pregnant. It should be emphasized that the administrator's criteria for defending conclusions from evaluation are not capricious, nor is there a great deal of freedom to choose which criteria will receive more weight at a given time. Like other standards for judging the quality of data or of inferences drawn from data, one is unable to maximize all criteria simultaneously. The administrator's choice is often situation-specific and
321
guided by the demands and requirements of stakeholder interests in a particular issue. As is evident from Table 1, administrators' justifications for drawing conclusions are more likely to be substantive than methodological. Cell 9 contains the only criteria that are solely concerned with epistemology and empirical methods. In order to predict the specific kinds of conclusions that management might favor, it is necessary to examine the purposes of brokering in a given situation. The columns in Table 1 indicate that administrators' goals can range from problem-framing (conceptual) or problem-solving (instrumental), to the need to justify a particular solution (symbolic). Considering the rows in Table 1, it is likely that administrators would have the most difficulty justifying conclusions from naturalistic inquiry when the brokering process was primarily one of program monitoring, since those data gathering activities are more easily satisfied with quantitative designs and measurement. Thus, cells 7, 8, and 9 in Table 1 present a greater challenge for management in terms of justifying conclusions from naturalistic inquiry. The framework can also be used by evaluators as a guide to the range of interpretive possibilities that could exist in a given study, although it does not identify the particular inquiry strategies that might be employed. In fact, from the perspective of management, it is the process by which data are interpreted that is crucial rather than the particular methodology by which data are generated (McClintock, 1987b). For instance, if the evaluation is designed for formative purposes, there are a variety of cognitive and group process exercises that might be used as heuristics, especially for conceptualizing and action processes, that would be of use to administrators (McClintock, 1987a; Moore, 1987). Given the greater number of substantive cells in Table 1 (cells 1-8), it is possible that naturalistic evaluations, with their emphasis on rich description, could be of more use to administrators than purely quantitative studies. Indeed, the framework in Table 1 reinforces the idea that, in terms of their brokering responsibilities, administrators themselves can be viewed as in-house ethnographers.
CONCLUSION All the balancing and hopping around do not yield a very neat or predictable answer to the question of how conclusions could be justified from naturalistic evalull am grateful to John Smith, of the University of Northern Iowa, for pointing out that the administrator's balancing or hopping should be conceptualized in a three-dimensional space, perhaps around a centered point of reference. Like Nagel's (1986) subjectiveself, this point could represent a particular goal, time frame or other subjectivelyvalued perspective around which reality is interpreted.
ation. These metaphors are meant to convey a sense of the dynamic complexity of the administrator's information brokering responsibilities, and the variety of criteria that might be used to make knowledge claims from the perspective of management. Though the application of particular criteria might be determined by situational expediency, Table 1 attempts to identify the range of possible justifications and to make explicit the basis by which knowledge claims are justified.
322
CHARLES McCLINTOCK
As noted, the job of the administrator as an information broker in organizational environments that are characterized by ambiguity and sense-making demands is similar to that of the naturalistic evaluator. Both roles are concerned primarily with processes by which information is interpreted and they require careful attention to multiple perspectives. The administrator may be more intermittently concerned with trustworthiness or other forms of technical adequacy than the evaluator, due to stakeholder pressure on substantive matters,
but this criterion often still will be important due to the pervasive cultural requirement that management behave as if it were rational. On the other hand, the action orientation of administrators will nearly always force them to act like realists. Within formal organizations it is the task of management to take a stand on which empirical conclusions most accurately reflect objective reality and will be most likely to achieve an objective, while some naturalistic evaluators and philosophers can still afford the luxury of being relativists.
REFERENCES ACKOFF, R. (1967). Management misinformation systems. Management Science, 14, 147-156. ALLISON, G.T. (1971). Essence o f decision: Explaining the Cuban missile crisis. Boston: Little-Brown. ARGYRIS, C., & SCHON, D.A. (1978). Organizational learning: A theory o f action perspective. Reading, MA: Addison-Wesley.
FELDMAN, M.S., & MARCH, J.G. (1981). Information in organizations as signal and symbol. Administrative Science Quarterly, 26, 171-186. GERGIN, K.J. (1985). The social constructionist movement in modern psychology. American Psychologist, 40, 266-275.
BENNIS, W., & NANUS, B. (1985), Leaders: The strategies for taking charge. New York: Harper and Row.
HASENFIELD, Y., & ENGLISH, R.A. (1974). Human service organizations: A conceptual overview. In Y. Hasenfield & R.A. English (Eds.), Human service organizations (pp. 1-23). Ann Arbor, MI: University of Michigan Press.
BICKMAN, I_.. (1987). The functions of program theory. In L. Bickman (Ed.), New directions f o r program evaluation: Using program theory in evaluation (pp. 5-18). San Francisco: Jossey-Bass.
HEDBURG, B.L.T., NYSTROM, D., & STARBUCK, W.H. (1976). Camping on seesaws: Prescriptions for a self-assessing organization. Administrative Science Quarterly, 21, 41-65.
BOYATZIS, R.E. (1985). The competent manager: A model f o r effective performance. New York: John Wiley & Sons.
LEVITON, L.D., & HUGHES, E.F. (1981). Research on the utilization of evaluation: A review and synthesis. Evaluation Review, 5, 525-548.
BRUNSSON, N. (1982). The irrationality of action and action rationality: Decisions, ideologies and organizational actions. Journal o f Management Studies, 19, 29-44.
LINCOLN, Y.S., & GUBA, E. (1986). Naturalistic inquiry. Beverly Hills, CA: Sage.
CHELIMSKY, E. (1987). The politics of program evaluation. In D.S. Cordray, H.S. Bloom, & R.L. Light (Eds.), Evaluation practice in review. New directions f o r program evaluation (pp. 5-22). San Francisco: Jossey-Bass. CLARK, D.L. (1985). Emerging paradigms in organizational theory and research. In Y.S. Lincoln (Ed.), Organizational theory and inquiry (pp. 43-78). Beverly Hills, CA: Sage. COHEN, M.D., MARCH, J.G., & OLSEN, J.P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17, 1-25. COOK, T.D. & CAMPBELL, D.T. (1979). Quasi-experimentation: Design and analysis issues f o r field settings. Skokie, IL: Rand McNally. CRONBACH, L.J. & ASSOCIATES. (1980). Toward reform o f program evaluation. San Francisco: Jossey-Bass. DONNELLON, A., GRAY, B., & BOUGON, M.G. (1986). Communication, meaning and organized action. Administrative Science Quarterly, 31, 43-55. EDEN, C., JONES, S., & SIMS, D. (1983). Messing about in problems: An informal structured approach to their identification and management. Elmsford, NY: Pergamon.
MARCH, J.G., & OLSEN, J.P. (1976). Ambiguity and choice in organizations. Bergen, Norway: Universitetsforlaget. McCASKEY, M.B. (1982). The executive challenge: Managing change and ambiguity. Marshfield, MA: Pitman. McCLINTOCK, C. (1985). Process sampling: A method for case study research on administrative behavior. Educational Administration Quarterly, 21,205-222. McCLINTOCK, C. (1986). Toward a theory of formative evaluation. In M.W. Lipsey & D.S. Cordray (Eds.), Evaluation studies review annual (pp. 205-223). Vol. 11. Beverly Hills, CA: Sage. McCLINTOCK, C. (1987a). Conceptual and action heuristics: Tools for the evaluator. In L. Bickman (Ed.), Using program theory in evaluation: New directions for program evaluation (pp. 43-57). San Francisco: Jossey-Bass. McCLINTOCK, C. (1987b). Managing information: The interpretation process. (Program Evaluation Paper Series, No. 9). Ithaca, NY: Department of Human Service Studies, Cornell University. McGOWAN, E.F. (1976). Rational fantasies. Policy Sciences, 7, 439-454. MINTZBERG, H. (1973). The nature o f managerial work. New York: Harper and Row.
Managerial Perspective MOORE, C.M. (1987). Group techniques for idea building. Beverly Hills, CA: Sage. NAGEL, S. (1986). The view from nowhere. New York: Oxford University Press. PATTON, M.Q. (1978). Utilization-based evaluation. Beverly Hills, CA: Sage. PHILLIPS, D.C. (1987). Philosophy, science and social inquiry. Elmsford, NY: Pergamon Press. SCHERIER, M.A. (1987). Program theory and implementation theory: Implications for evaluators. In L. Bickman (Ed.), Using program theory in evaluation: New directions for program evaluation (pp. 5976). San Francisco: Jossey-Bass. SCHON, D.A. (1979). Generative metaphor: A perspective on problem-setting in social policy. In A. Ortony (Ed.), Metaphor and thought (pp. 254-283). Cambridge, England: Cambridge University Press. SIMS, H.P., JR., GIO1A, D.A., & ASSOCIATES. (1986). The thinking organization: Dynamics of organizational social cognition. San Francisco: Jossey-Bass.
323
STUFFLEBEAM, D.L. (1983). The CIPP model for program evaluation. In G.F. Madaus, M. Scriven, & D.C. Stufflebeam (Eds.), Evaluation models (pp. 117-142). Boston: Kluwer-Nijhoff. STRASSMAN, P.A. (1985). Information payoff: The transformation of work in the electronic age. New York: The Free Press. TROCHIM, W.M.K., & LINTON, R. (1986). Conceptualization for planning and evaluation. Evaluation and Program Planning, 9, 289308. WEICK, K.E. (1977). Repunctuating the problem. In P.S. Goodman & J.M. Pennings (Eds.), New perspectives in organizational effectiveness (pp. 193-225). San Francisco: Jossey-Bass. WEICK, K.E. (1979). The social psychology of organizing. Reading, MA: Addison-Wesley. WEICK, K.E. (1985). Sources of order in underorganized systems: Themes in recent organizational theory. In Y.S. Lincoln (Ed.), Organizational theory and inquiry (pp. 106-136). Beverly Hills, CA: Sage. WHETTEN, D.A. (1984). Effective administrators: Good management on the college campus. Change, November-December, 38-43.
0149-7189/87 $3.00 + .00 Copyright © 1987 Pergamon Journals Ltd
Printed in the USA. All rights reserved.
ADMINISTRATORS AS INFORMATION BROKERS A Managerial Perspective on Naturalistic Evaluation
CHARLES McCLINTOCK Cornell University
ABSTRACT The managerial perspective on justifying conclusions f r o m naturalistic evaluation is based on how administrators function as brokers who exchange, highlight, and interpret information f o r others. Information brokering is particularly important within a cognitive perspective on organizations, and consists o f process and outcome dimensions. Brokering processes include using evaluative information f o r conceptualizing, motivating action, and monitoring performance. Brokering outcomes refer to conceptual, instrumental, and symbolic results o f information use. The combination o f these dimensions results in a variety o f substantive and methodological criteria that administrators might use to justify knowledge claims f r o m program evaluation.
guide the design, data collection, or analysis of any evaluation, regardless of its epistemological paradigm. This approach to justifying evaluation conclusions is based on a theoretical framework for formative evaluation developed by McClintock (1986). The framework is based on three components: methodology, program theory, and evaluation context that, taken together, provide guidance for the design and analysis of evaluation efforts. The latter two categories are of particular importance for evaluation design and analysis, and consist of substantive issues having to do with the theory that underlies program activities and their expected effects, and the contexts (e.g., community, organizational) that would influence program implementation (see also Bickman, 1987; Scherier, 1987). This framework contrasts with perspectives on evaluation that are solely or predominately methodological (e.g., Cook & Campbell, 1979; Lincoln & Guba, 1985), and whose criteria for justifying conclusions would be epistemological. The approach is similar to Patton's (1978) utilization-focused approach to evaluation, Stufflebeam's (1983) decision-oriented C I P P model, and to Cronbach's (1980, Chapters 2 and 4) position, in particular his discussion of the interests of varied evaluation users in the "policy shaping community." Briefly, the latter perspective highlights evaluation users' (e.g., program
Administrators often serve their organizations as resident ethnographers. Their jobs require them to content analyze communications, have goal-based conversations that are like semi-structured interviews, and sit as participant observers through a constant stream of meetings. The information, inferences, and conclusions drawn from these activities are then conveyed and exchanged among various constituencies inside and outside the organization. It is this process of dealing in and negotiating interpretations of reality that qualifies the administrator as an information broker, and in an informal sense, as a naturalistic evaluator. At the same time administrators often are required to draw conclusions from formal evaluation studies. As users of evaluation data, administrators would justify knowledge claims with reference to the same criteria that they use as informal ethnographers. These criteria are largely determined by administrators' role responsibilities as information brokers. In this paper a conceptual framework will be developed that describes how administrators function as brokers for various types of information as a means of developing, maintaining, or improving programs. From this framework a set of managerial questions and concerns are derived that determine the acceptability of knowledge claims from naturalistic inquiry, and that also could be used to
Requests for reprints should be sent to Charles McClintock, Department of Human Service Studies, College of Human Ecology, Cornell University, Ithaca, NY 14853. 315
316
CHARLES McCLINTOCK
administrators, operating personnel in local sites) distinct information needs and concerns, that in turn are dependent on the type of program and its stage of development. These substantive, as opposed to methodological, concerns largely determine the relevance and validity of evaluation conclusions. Although not the central question in this paper, one might ask how managers would prefer to have evaluatots warrant knowledge claims. The managerial perspective on this question is well stated by Chelimsky THE EPISTEMOLOGICAL
(1987) and Phillips (1987), both of whom cogently argue for the distinctive integrity of scientific method as a process in which knowledge claims are publicly and systematically tested against observation. To the extent that naturalistic inquiry emphasizes relativism and multiple realities, it is less useful to administrators who already are too familiar with that dilemma, and who seek in an evaluator an independent and intentionally (not absolutely) objective party.
POSITION(S) OF MANAGEMENT
Managers are neither scientists nor philosophers, but as practicing information brokers their work bears a resemblance to applied science and philosophy, and thus there are at least implicit epistemological underpinnings for administrative behavior. From the managerial perspective, evaluation conclusions are warranted to the extent that they are consistent with the information brokering requirements of the administrator's role. This pragmatic view would apply regardless of whether or not an evaluation were done within the naturalistic paradigm, and thus it pertains to any systematic empirical product presented to program managers for their use. This position presents a problem philosophically, however, since administrators have to expect to "play the epistemological field" due to the breadth of their information brokering responsibilities. There are three general brokering responsibilities that imply a range of epistemological positions by which managers could justify evaluation conclusions. First, administrators are expected to represent, negotiate, and possibly synthesize the varying perceptions of different program stakeholders. This position emphasizes relativism and the validity of minority as well as majority viewpoints. More importantly, administrators often act like constructionists since they are expected to manage a process in which reality is newly synthesized from the social interplay among various parties to the issue at hand (Gergin, 1985). Second, management often needs data to make short term decisions and adjustments to program activities. Administrators try to step back from the stream of events as i f reality were objective, and to experiment with it in a search for activities that will yield the most desired results. This strategy assumes an epistemology of convergent or fallible realism in which the administrator uses information to obtain a more accurate, though always incomplete or biased, picture of reality that requires continued exploration and revision (Argyris & Sch6n, 1978). Finally, administrators are often called upon to take a position and justify it as if there were no viable alternative version of reality. As leaders with vision or enforcers of the power of a dominant coalition, this managerial requirement results in the administrator as a realist in which
conditions of the world can be known objectively without uncertainty. Paradoxically, it may often be the case that the administrator as realist only uses data symbolically to justify a preference or to signal that a correct or rational decision has been made (Feldman & March, 1981). Under these circumstances evaluation conclusions are warranted when they can be defended as scientifically rational, and when they support a position already endorsed by the administrator or dominant coalition. Despite the range of epistemologies that administrators might use to justify conclusions from evaluation findings, they are most often forced to act like fallible realists, with the emphasis on the latter term. That is, in many situations an administrator must make a choice among several worthy, persuasive, but competing alternatives about what course of action to take. Often that choice will be based on a belief that a particular set of recommendations more accurately reflects reality and will be more likely to achieve an objective. The administrator will often be the first to admit that objectives can shift and that reality can change quickly or be imperfectly and variably assessed, but in the meantime it is often necessary to take a stand. In fact, the popular and research-based literature on management and leadership consistently identifies having a strong vision of reality and a commitment to action as key attributes of the effective administrator (Bennis & Nanus, 1985). The terms fallible or convergent are important too, however, since they are consistent with the prescription that a good administrator tries to listen to different points of view, and indeed almost of definition, is often required to represent the views of all the relevant stakeholders (Whetten, 1984). From the perspective of management there is truth and importance in various versions of organizational life, and the wise administrator tries to acknowledge some validity in all or many of them, recognizing that multiple subjectivity is part of the job. This pragmatic position of management is similar to what Nagel (1986) described as a possibly irreconcilable dualism between objective and subjective dynamics in
Managerial Perspective philosophy. He contended that scientific and technological truth depend on processes that seek to develop knowledge of an objective reality, independent of the particular experiences and perspectives of individual judgement. At the same time, the subjectivity of self is a part of that reality, and knowledge of the world
317
is embodied in individual minds. Justifying conclusions from a managerial perspective requires evaluation methods that strive for an objective (though never finally so) view of the world, that also includes the effects of subjective interpretations of reality.
C O G N I T I O N A N D I N F O R M A T I O N IN O R G A N I Z A T I O N S The idea that administrators, as fallible realists, largely act as brokers of information is particularly significant within the cognitive perspective on organizational behavior (Clark, 1985; Weick, 1979). In this view a critical unit of activity is the sense-making process in which small, often highly segmented groups of individuals try to create sensible and overlapping interpretations of their environment. Sense-making is a process that does not require consensus of interpretation. Dissimilar interpretations can have "equifinality of meaning" when they have compatible implications for action (Donnellon, Gray, & Bougon, 1986). For example, the "Star Wars" defense initiative might be endorsed by conservatives and liberals for very different reasons: the former because they believe it will forestall any arms negotiations with the Soviet Union, and the latter because they believe it can be used as a bargaining chip to achieve massive arms reduction. The importance of sense-making as an organizational process depends on the degree to which ambiguity is characteristic of the work tasks. Ambiguity is the product of the following kinds of conditions: (a) perceptions of problems and solutions are varied and often not explicit, (b) service outcomes are governed more by trial and error than by planned action, (c) the nature of the problems or work is often unclear or changing, (d) goals and causal processes are subject to change and multiple interpretation, and (e) objective standards for evaluation are generally lacking, forcing reliance on socially determined or value-based criteria for judging success (McCaskey, 1982; Weick, 1985). From this perspective, administrators, as well as other professionals, are expected to deal creatively with organizational situations that look like anarchies (March & Olsen, 1976), garbage cans, (Cohen, March, & Olsen, 1972), seesaws (Hedburg, Nystrom, & Starbuck, 1976), and octopoids (Weick, 1977). These metaphors reinforce the importance of information and cognition in organizations, and highlight administrative responsibilities such as sense-making and the interpretation of cause maps, schemas, scripts, symbols, and myths (Sims, Gioia, & Associates, 1986, pp. 1-19). The Environment of H u m a n Service Tasks
The concepts of cognition, ambiguity, and sense-making are especially significant for human service settings in which program evaluation is likely to be prevalent.
Human services refer to those activities whose purpose is to " . . . maintain or improve the general well-being and functioning of people" (Hasenfield & English, 1974, p. 1). While it is possible to classify entire organizations as human services (e.g., schools, hospitals, social service agencies), many types of organizations contain human service activities. For example, administrative, marketing, personnel, planning, training, and internal audit roles, with their emphasis on serving others, can be thought of as embodying human service responsibilities, regardless of the organizational setting (Eden, Jones, & Sims, 1983). Hasenfield and English (1974, pp. 1-32) described several distinctive attributes of human service tasks that highlight the importance of cognitive processes. First, human service professionals work with human beings as their "raw material." Interpersonal perception, moral judgment, social norms and status, and client motivation and autonomy are critical to the effectiveness of service treatments, and to relations between professionals and clients. Second, the goals in human services are often valueladen, ambiguous, and conflicting. For example, approaches to the treatment of juvenile delinquency will be seen by different segments of society as serving such varied purposes as personal growth, reform, control and prevention, or substitution for familial care and responsibility. A given human service encounter may also reflect the differing goals of client, professional, and organization, and a mixture of norms and standards from accrediting, advocacy, professional, or ideological groups. Finally, the "technology" of human service treatments or tasks is often indeterminant, due to variability in human characteristics, motivations, and moods, the clarity of the outcomes that are sought, and the state of theory and knowledge about cause-effect relationships that will produce desired change. To summarize, organizations are thought of as systems for communicating and negotiating the varied, often ambiguous, cognitions and perceptions of reality of groups of stakeholders. Especially in human service settings, interpreting information is a key administrative responsibility. In order to understand better the managerial perspective on drawing conclusions from naturalistic studies, it is necessary to look more closely at how administrators manage information and its interpretation.
318
CHARLES McCLINTOCK INFORMATION
If organizations are viewed correctly as systems of information and cognition, then information can be seen as an important resource to manage. At a macrolevel there is growing recognition that information can be related to improvements in organizational effectiveness, but relatively little is known about the economics and social psychology of information management (Strassman, 1985). Managing information requires attention to several broad areas of administrative responsibility including the following: (a) the acquisition and maintenance of information hardware and software, (b) the design and
MANAGEMENT administration of information systems, (c) staff training, and (d) the utilization of information for program planning, reporting, and management. The emphasis here is on the social psychology o f the utilization process, and in particular on the brokering aspect of utilization; that is, how information is exchanged and interpreted by administrators in order to structure perceptions, motivate action and justify managerial performance. Other important dimensions of utilization include managing forums for information exchange, and the cognitive aspects of causal inference (McClintock, 1987b).
BROKERING INFORMATION USE Administrators are often required to use information in such a way that their programs and decision processes look orderly and rational (Feldman & March, 1981; McGowan, 1976). This situation arises from the expectations of external stakeholders as a prerequisite for accountability, funding, and political support, and of internal stakeholders who often have a need to impose order and rational explanation on events (Weick, 1985). In many instances, however, by virtue of their position in the organization, it is administrators more than anyone else who have a true sense of the fragmented and unplanned course of events, and the variable perceptions of different stakeholders. Thus, information may be used to inform two or more individuals when an opportunity is seen to connect otherwise uncoupled but complementary actions, or to provide new perspectives to groups whose attention and access to information are more narrow than that of the administrator. This description of administrative responsibility, as well as others grounded in behavioral observations (Mintzberg, 1973), conveys an image of the administrator as a broker among multiple, autonomous and often adversarial constituencies inside and outside the organization's boundaries. Information brokering is inherent in the roles and responsibilities of management. Whetten (1984), for example, described several of the information brokering demands on the administrator in higher education. He argued that effective management requires attention to (a) coalition building in which constituent needs and interests are continuously assessed and interrelated, (b) dissemination of new ideas and opportunities, (c) personification of institutional image, (d) interpreting the results of planning and evaluation in a visionary manner, and (e) "over-communicating" by maintaining two-way information flow especially in times of stress or retrenchment. Boyatzis (1985) identified nineteen competencies related to effective managerial performance across 41 different administrative jobs in public and private sector
organizations. Of the 19, the following 10 skills are directly related to the concept o f information brokering; proactivity in information seeking and action, diagnostic use of concepts to find patterns, use of oral presentations, logical thought, conceptualization, social network building, managing group process, perceptual objectivity, concern with close relationships, and ability to convey relevant knowledge. As a broker, the administrator acts as an agent for others by exchanging information among them, alerting them to opportunities, constraints, and interpretations that they might not be aware of, and creating settings and incentives for information to be exchanged, debated, interpreted, and acted on. Administrators may be brokers for many things including cognitions, feelings, power, goals, resources, structural arrangements, and personnel decisions, but the common medium of their work is information. A fundamental responsibility of the human service administrator is to broker information in order to cope with ambiguity, and facilitate understanding and action in the organization. As an information broker, the administrator is faced with the following problems: 1. Information overload and fragmentation. Often there is too much information in too many different forms. The task here is to find ways of processing, relating and conceptualizing information so that it can be absorbed and meaning extracted from it. Organizations have few formal tools, forums, or incentives for coping with information overload and fragmentation (Ackoff, 1967). 2. Relating information to the constraints and social psychological requirements for action. Often information is not timely or in a format that reflects the programmatic structure around which choices and action have to be taken. For example, information in budgets is often not useful for guiding the efforts of personnel due to the prevalence of line item budgeting over program budgeting. By making it difficult to re-
Managerial Perspective late the allocation of equipment, staff or operating funds to desired goals and outcomes, one complicates the perception of others and thus limits the power of the information to focus their attention, commitment, and motivation around programmatic efforts. 3. Having information available on critical indicators of program implementation. Programs may succeed or fail based on the soundness of their underlying theory or because of the ways they are implemented. It is important to establish a monitoring process that allows the organization to assess whether and to what extent a program has been implemented, and to identify those processes that are believed to be critical to its success. These problems establish some of the basic information brokering responsibilities of the administrator, which can be summarized as follows: (a) minimize overload and fragmentation by conceptualizing or framing problems, decisions, and programs in different ways; (b) focus attention on a few decision and action alternatives; (c) monitor the processes of program implementation. In pursuit of these responsibilities, administrators must rely on a variety of informal information sources, such as grapevines, pipelines and myths, and on technical specialists, including evaluators, to monitor information thresholds, and fill information gaps (McClintock, 1985). To summarize, as brokers of information, administrators become spokespersons for the perceptions of others as well as themselves, and they often try to find common ground that will facilitate action. The essence of this process is to use, and govern the use of, language and other symbols, stimulate action from which information about reality will be produced, facilitate interactions among stakeholders in order to share perceptions, and conduct retrospective interpretations of events. The sum of these activities is the management of meaning which is a core responsibility of the administrator's information and communication responsibilities (Bennis & Nanus, 1985, pp. 110-151; Weick, 1985, pp. 132-133).
A Framework for Information Brokering With this general background to information brokering, it is possible to identify a process and an outcome dimension for the concept. The process dimension consists of three distinct but interrelated parts, which can be described as follows: 1. The conceptualizing process, in which large amounts of disparate information are clustered to identify, define, or prioritize problems, processes, opportunities, and past events (Trochim & Linton, 1986). Often the administrator will want to combine perspectives to find new and creative ways of defining reality in order to break deadlocks a m o n g unreconciled perceptions held by others (Sch6n, 1979), or to contrast perspectives that highlight varying assumptions, issues and out-
319
comes (Allison, 1971). The dynamic of conceptualizing is to create, expand or synthesize perceptions of reality. 2. The action process, in which information is used to motivate behavior towards specific decision alternatives, or to create sufficient energy and commitment to implement a decision. The social psychological requirements of concluding a decision or motivating action often mean that reality must be narrowed or simplified, and attention directed to a s!ngle alternative, or to specific standard operating procedures, goals and constraints (Brunsson, 1982). The dynamic of the action process is to narrow the focus of attention to information on one or two alternative choices. 3. The monitoring process, in which one attempts to learn about the effects of chosen decisions, actions or programs. There are two important parts to this activity. First, it is necessary to know whether the p r o g r a m has been implemented and to what extent it is consistent with intended policy. A second process involves making causal inferences about program effects and distinguishing effects expected from the theory that underlies the program from those due to its implementation in a particular agency (Scherier, 1987). The dynamic of the monitoring process is one in which information is used to compare, confirm, or question expectations about human service problems and programs. These are the three components of the process dimension of information brokering. The other dimension of information brokering is defined in terms of the outcomes of information use. As described by Leviton and Hughes (1981), information use can result in the following outcomes: 1. A conceptual outcome where information users gain new understanding of a problem or program. When perceptions are deadlocked, administrators will favor evaluation conclusions that reframe the situation in such a way that perceptual deadlocks are overcome (Donnellon et al., 1986). 2. An instrumental outcome in which information is used to narrow the alternatives, motivate action and achieve short term objectives. 3. A symbolic outcome in which information persuades others about a particular interpretation, or indicates that careful or rational analysis has been undertaken, whether or not it has. Often administrators will favor conclusions that have the strongest scientific character, since these embody the norms of rationality that govern managerial responsibilities. There is some parallelism between these three information brokering outcomes and the three brokering processes presented earlier (i.e., between conceptual outcomes and conceptualizing processes, or instrumental outcomes and action processes). For the present, however, processes and outcomes are conceived as independent dimensions of information brokering. By
C H A R LES McCLINTOCK
320
crosstabulating them in a matrix it is possible to derive a set of questions and concerns that identify specific information brokering responsibilities and requirements, JUSTIFYING CONCLUSIONS Table 1 displays the crosstabulation of brokering processes and outcomes. Illustrative questions and concerns are shown in each cell that define the possible range of an administrator's information brokering responsibilities. Knowledge claims and conclusions would be acceptable to the administrator to the degree that they were consistent with those information brokering responsibilities that were most salient. Based on Table 1 it appears that a broad range of claims and conclusions from naturalistic evaluation could be justified from the managerial perspective. In practice, however, not all cells in the matrix would have equal importance in the context of a particular study. Two metaphors for the role requirements of the administrator can be used to describe how some cells might be weighted more than others. The first metaphor is to think of administrators as high-wire performers (without a safety net below), in that they often are trying to achieve a balance among a set of gravitational forces that are constantly tipping them one direction or another. In terms of Table 1, this image of managerial activity suggests that an administrator will often be balancing the demands of a particular column, row or diagonal. For example, if the program being evaluated were new and controversial
and therefore that would be used by administrators to justify evaluation conclusions.
FROM E V A L U A T I O N
STUDIES
(e.g., making contraceptives available in the schools), the demands of instrumental and symbolic use would be most pressing due to the need for quick action on a politically volatile issue. Management could easily find itself trying to balance the requirements of cells 3, 5, and 9 in the second and third columns of Table 1. Depending on personality characteristics, organizational routines and traditions, and the particular patterns of pressure from others, the administrator could try to balance in the middle and endorse conclusions from cell 5 (the action process), or tilt towards the conclusive value of information in cells 3 or 9 (the conceptual or monitoring processes). In the example of contraceptives in the schools it is likely that the balancing act would be between cells 3 and 5 due to the immediacy of the problem and the intensity of feelings about issues of teenage pregnancy, sexuality and contraception. As noted earlier, the information requirements for action are different than for conceptualizing. To accomplish a decision and motivate action usually means using information to simplify reality in order to minimize cognitive demands and maximize commitment and energy for action. The purpose is to arrive at a single alternative, model or story that stimulates and rewards action. Conceptualizing, on the other hand, re-
TABLE 1 A FRAMEWORK FOR INFORMATION BROKERING IN ORGANIZATIONS Brokering Outcomes Brokering Processes
Conceptualizing
Conceptual
Instrumental
Symbolic
Are perceptual deadlocks broken? Are new concepts, theories, problems, questions or plans discovered?
Is a problem discovered that allows a latent solution to be considered?
Does the information persuade and help gain political or financial support?
Is new thinking generated about the problem or possible solutions?
Does it conform to the values and schema of different stakeholders?
1 Are costs, risks, or benefits redefined or disovered?
Does the information help motivate staff, clients, or others to reach a decision or direct their efforts in a particular direction?
Does the information help justify or explain resource allocation decisions?
Are new behavior patterns, methods of implementation, barriers, or opportunities discovered?
Is the information detailed, timely, and reflective of the complexity and variety of programs as they are being implemented?
Is the information defensible as systematic, trustworthy, or scientific?
Action
Monitoring
3
7
8
9
Managerial Perspective quires information to be taken through many cycles of manipulation and interpretation, and multiple stories are generated in order to acknowledge the perceptions and concerns of as m a n y stakeholders as possible. Weick (1985, p. 130) summarizes this tradeoff between the information demands of action versus conceptualizing as follows: It is relatively hard to gain the best of both worlds. Administrators run the risk of knowing their world well but being unable to do anything about it, or they run the opposite risk of making some decisive change in the world only to discover that they changed the wrong thing and the problem got worse. The second metaphor is to think of the administrator as a frog in a lily pond, hopping to one lily pad and sitting for a while, then to the next, and so on. Sitting is not done, however, to relax and enjoy the sunshine, but rather to satisfy the needs or put out the fires of one stakeholder group while deferring others, eventually trying to get to as many places as possible but in an alternating or sequential fashion. Thus, staying with the contraception example, it might be necessary to: (a) hop to cell 1 in order to find a creative way of thinking of the problem (e.g., define teenage pregnancy as a public health issue, not as a matter of sexuality); (b) then to cell 9 to bring to bear the leverage of systematic, trustworthy, or scientific evidence on the seriousness of the situation; (c) then some quick hops between cell 3, to develop varied portrayals of the program that will calm the fears and attend to the concerns of the school board, PTA, students, religious groups and human service professionals, and cell 4 to check with the lawyers on the legal liability of the school in cases where a student who receives contraceptives subsequently gets pregnant. It should be emphasized that the administrator's criteria for defending conclusions from evaluation are not capricious, nor is there a great deal of freedom to choose which criteria will receive more weight at a given time. Like other standards for judging the quality of data or of inferences drawn from data, one is unable to maximize all criteria simultaneously. The administrator's choice is often situation-specific and
321
guided by the demands and requirements of stakeholder interests in a particular issue. As is evident from Table 1, administrators' justifications for drawing conclusions are more likely to be substantive than methodological. Cell 9 contains the only criteria that are solely concerned with epistemology and empirical methods. In order to predict the specific kinds of conclusions that management might favor, it is necessary to examine the purposes of brokering in a given situation. The columns in Table 1 indicate that administrators' goals can range from problem-framing (conceptual) or problem-solving (instrumental), to the need to justify a particular solution (symbolic). Considering the rows in Table 1, it is likely that administrators would have the most difficulty justifying conclusions from naturalistic inquiry when the brokering process was primarily one of program monitoring, since those data gathering activities are more easily satisfied with quantitative designs and measurement. Thus, cells 7, 8, and 9 in Table 1 present a greater challenge for management in terms of justifying conclusions from naturalistic inquiry. The framework can also be used by evaluators as a guide to the range of interpretive possibilities that could exist in a given study, although it does not identify the particular inquiry strategies that might be employed. In fact, from the perspective of management, it is the process by which data are interpreted that is crucial rather than the particular methodology by which data are generated (McClintock, 1987b). For instance, if the evaluation is designed for formative purposes, there are a variety of cognitive and group process exercises that might be used as heuristics, especially for conceptualizing and action processes, that would be of use to administrators (McClintock, 1987a; Moore, 1987). Given the greater number of substantive cells in Table 1 (cells 1-8), it is possible that naturalistic evaluations, with their emphasis on rich description, could be of more use to administrators than purely quantitative studies. Indeed, the framework in Table 1 reinforces the idea that, in terms of their brokering responsibilities, administrators themselves can be viewed as in-house ethnographers.
CONCLUSION All the balancing and hopping around do not yield a very neat or predictable answer to the question of how conclusions could be justified from naturalistic evalull am grateful to John Smith, of the University of Northern Iowa, for pointing out that the administrator's balancing or hopping should be conceptualized in a three-dimensional space, perhaps around a centered point of reference. Like Nagel's (1986) subjectiveself, this point could represent a particular goal, time frame or other subjectivelyvalued perspective around which reality is interpreted.
ation. These metaphors are meant to convey a sense of the dynamic complexity of the administrator's information brokering responsibilities, and the variety of criteria that might be used to make knowledge claims from the perspective of management. Though the application of particular criteria might be determined by situational expediency, Table 1 attempts to identify the range of possible justifications and to make explicit the basis by which knowledge claims are justified.
322
CHARLES McCLINTOCK
As noted, the job of the administrator as an information broker in organizational environments that are characterized by ambiguity and sense-making demands is similar to that of the naturalistic evaluator. Both roles are concerned primarily with processes by which information is interpreted and they require careful attention to multiple perspectives. The administrator may be more intermittently concerned with trustworthiness or other forms of technical adequacy than the evaluator, due to stakeholder pressure on substantive matters,
but this criterion often still will be important due to the pervasive cultural requirement that management behave as if it were rational. On the other hand, the action orientation of administrators will nearly always force them to act like realists. Within formal organizations it is the task of management to take a stand on which empirical conclusions most accurately reflect objective reality and will be most likely to achieve an objective, while some naturalistic evaluators and philosophers can still afford the luxury of being relativists.
REFERENCES ACKOFF, R. (1967). Management misinformation systems. Management Science, 14, 147-156. ALLISON, G.T. (1971). Essence o f decision: Explaining the Cuban missile crisis. Boston: Little-Brown. ARGYRIS, C., & SCHON, D.A. (1978). Organizational learning: A theory o f action perspective. Reading, MA: Addison-Wesley.
FELDMAN, M.S., & MARCH, J.G. (1981). Information in organizations as signal and symbol. Administrative Science Quarterly, 26, 171-186. GERGIN, K.J. (1985). The social constructionist movement in modern psychology. American Psychologist, 40, 266-275.
BENNIS, W., & NANUS, B. (1985), Leaders: The strategies for taking charge. New York: Harper and Row.
HASENFIELD, Y., & ENGLISH, R.A. (1974). Human service organizations: A conceptual overview. In Y. Hasenfield & R.A. English (Eds.), Human service organizations (pp. 1-23). Ann Arbor, MI: University of Michigan Press.
BICKMAN, I_.. (1987). The functions of program theory. In L. Bickman (Ed.), New directions f o r program evaluation: Using program theory in evaluation (pp. 5-18). San Francisco: Jossey-Bass.
HEDBURG, B.L.T., NYSTROM, D., & STARBUCK, W.H. (1976). Camping on seesaws: Prescriptions for a self-assessing organization. Administrative Science Quarterly, 21, 41-65.
BOYATZIS, R.E. (1985). The competent manager: A model f o r effective performance. New York: John Wiley & Sons.
LEVITON, L.D., & HUGHES, E.F. (1981). Research on the utilization of evaluation: A review and synthesis. Evaluation Review, 5, 525-548.
BRUNSSON, N. (1982). The irrationality of action and action rationality: Decisions, ideologies and organizational actions. Journal o f Management Studies, 19, 29-44.
LINCOLN, Y.S., & GUBA, E. (1986). Naturalistic inquiry. Beverly Hills, CA: Sage.
CHELIMSKY, E. (1987). The politics of program evaluation. In D.S. Cordray, H.S. Bloom, & R.L. Light (Eds.), Evaluation practice in review. New directions f o r program evaluation (pp. 5-22). San Francisco: Jossey-Bass. CLARK, D.L. (1985). Emerging paradigms in organizational theory and research. In Y.S. Lincoln (Ed.), Organizational theory and inquiry (pp. 43-78). Beverly Hills, CA: Sage. COHEN, M.D., MARCH, J.G., & OLSEN, J.P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17, 1-25. COOK, T.D. & CAMPBELL, D.T. (1979). Quasi-experimentation: Design and analysis issues f o r field settings. Skokie, IL: Rand McNally. CRONBACH, L.J. & ASSOCIATES. (1980). Toward reform o f program evaluation. San Francisco: Jossey-Bass. DONNELLON, A., GRAY, B., & BOUGON, M.G. (1986). Communication, meaning and organized action. Administrative Science Quarterly, 31, 43-55. EDEN, C., JONES, S., & SIMS, D. (1983). Messing about in problems: An informal structured approach to their identification and management. Elmsford, NY: Pergamon.
MARCH, J.G., & OLSEN, J.P. (1976). Ambiguity and choice in organizations. Bergen, Norway: Universitetsforlaget. McCASKEY, M.B. (1982). The executive challenge: Managing change and ambiguity. Marshfield, MA: Pitman. McCLINTOCK, C. (1985). Process sampling: A method for case study research on administrative behavior. Educational Administration Quarterly, 21,205-222. McCLINTOCK, C. (1986). Toward a theory of formative evaluation. In M.W. Lipsey & D.S. Cordray (Eds.), Evaluation studies review annual (pp. 205-223). Vol. 11. Beverly Hills, CA: Sage. McCLINTOCK, C. (1987a). Conceptual and action heuristics: Tools for the evaluator. In L. Bickman (Ed.), Using program theory in evaluation: New directions for program evaluation (pp. 43-57). San Francisco: Jossey-Bass. McCLINTOCK, C. (1987b). Managing information: The interpretation process. (Program Evaluation Paper Series, No. 9). Ithaca, NY: Department of Human Service Studies, Cornell University. McGOWAN, E.F. (1976). Rational fantasies. Policy Sciences, 7, 439-454. MINTZBERG, H. (1973). The nature o f managerial work. New York: Harper and Row.
Managerial Perspective MOORE, C.M. (1987). Group techniques for idea building. Beverly Hills, CA: Sage. NAGEL, S. (1986). The view from nowhere. New York: Oxford University Press. PATTON, M.Q. (1978). Utilization-based evaluation. Beverly Hills, CA: Sage. PHILLIPS, D.C. (1987). Philosophy, science and social inquiry. Elmsford, NY: Pergamon Press. SCHERIER, M.A. (1987). Program theory and implementation theory: Implications for evaluators. In L. Bickman (Ed.), Using program theory in evaluation: New directions for program evaluation (pp. 5976). San Francisco: Jossey-Bass. SCHON, D.A. (1979). Generative metaphor: A perspective on problem-setting in social policy. In A. Ortony (Ed.), Metaphor and thought (pp. 254-283). Cambridge, England: Cambridge University Press. SIMS, H.P., JR., GIO1A, D.A., & ASSOCIATES. (1986). The thinking organization: Dynamics of organizational social cognition. San Francisco: Jossey-Bass.
323
STUFFLEBEAM, D.L. (1983). The CIPP model for program evaluation. In G.F. Madaus, M. Scriven, & D.C. Stufflebeam (Eds.), Evaluation models (pp. 117-142). Boston: Kluwer-Nijhoff. STRASSMAN, P.A. (1985). Information payoff: The transformation of work in the electronic age. New York: The Free Press. TROCHIM, W.M.K., & LINTON, R. (1986). Conceptualization for planning and evaluation. Evaluation and Program Planning, 9, 289308. WEICK, K.E. (1977). Repunctuating the problem. In P.S. Goodman & J.M. Pennings (Eds.), New perspectives in organizational effectiveness (pp. 193-225). San Francisco: Jossey-Bass. WEICK, K.E. (1979). The social psychology of organizing. Reading, MA: Addison-Wesley. WEICK, K.E. (1985). Sources of order in underorganized systems: Themes in recent organizational theory. In Y.S. Lincoln (Ed.), Organizational theory and inquiry (pp. 106-136). Beverly Hills, CA: Sage. WHETTEN, D.A. (1984). Effective administrators: Good management on the college campus. Change, November-December, 38-43.