- Email: [email protected]
Performance measurement: Some conceptual issues
Evaluation andProgram Planning, Vol. 3, pp. 35-43.1980
0147-7189/80/010035-09$02.00/O Copyright 0 1980PergamonPress Ltd
Printedin the U.S.A. Ail rightsreserved.
PERFORMANCE
MEASUREMENT:
SOME CONCEPTUAL
ISSUES
TERRY CONNOLLY and STUART JAY DEUTSCH Georgia Institute of Technology
ABSTRACT Measures of how well a system is operating are clearly of interest to a wide range of users, from organizational researchers to system managers. The current literatures in such areas as ‘performance measurement, ” “organizational effectiveness, ” and the like show little consistency in their definitions of terms or in their methods for generating measures. The present paper attempts a clarification of these issues in the form of a conceptual minimalist position which requires only three basic definitions, and leaves, asfar as possible, all remaining issues open to empirical investigation. We first review the literature on “organizational effectiveness” contrasting the organizational goals and systems paradigms, and note the lack of either theoretical or empirical convergence between the two. An examination of the nature of effectiveness statements suggests that this failure of convergence flows mainly from the different criterion sets generated by the two paradigms - and, importantly, that one should not expect convergence on a single measure or set of measures which uniquely define how well a.system is performing. One’s view of how wella given system isperforming is a function of where onestands (either theoretically or in relationship to the system), andpursuit of the one true set of performance measures is a futile exercise. Instead, we propose to redirect attention to the identification of the various individuals and groups (‘constituencies”) with an interest in system performance, and to the investigation of those items of system relevant information (their “performance measure sets”‘) which do, in fact, change their evaluations of how well the system is performing. This perspective will, we hope, redirect effort from futile theoretical debate to empirical investigations of what measures are used, by whom, and to what effect, in specific settings.
INTRODUCTION cern in this article is with performance measures in this more restrictive sense, as indicators of ho% well an organization or system is functioning, that is, system effectiveness. Such indicators are, clearly, of interest to a wide range of potential users - managers of complex systems, researchers, innovators who attempt to bring about change in system functioning, and others. This diversity of users is reflected in the diversity of definitions of performance measures, and of methods of generating them. Researchers concerned with system performance frequently measure very different things,
There appears to be no satisfactory general definition for the term “performance measure” as applied to a complex system such as an educational system, an economic system, or a criminal justice system. In the broad sense, the term could be applied to any measure which reflects some aspect of system activity - that is, to any measure which addresses the question: “How is the system performing?” The term is more frequently used in a narrower sense to denote a measure of the success, effectiveness, or desirability of the system’s behavior - that is, a measure which addresses the question: “How well is the system performing?” Our con-
This work was performed under grant number 78-NI-AX-0003 from the National Institute of Law Enforcement and Criminal Justice. Points of view or opinions stated in this document are those of the authors and do not necessarily represent the official positions or policies of the U.S. Department of Justice Requests for reprints should be sent to Terry Connolly, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332.
35
TERRY CONNOLLY
and STUART J.4Y DEUTSCH
and in very different ways.. Comparison of their findings is thus often hazardous, if not downright impossible. This article attempts to bring some conceptual clarification to the area. The first part of the article reviews the several definitions and approaches to performance measurement which have been offered by researchers concerned with the assessment of the effec-
tiveness or performance of one class of complex systems, the single organization. The second part of the article considers more closely just what is involved in making a statement about the effectiveness or performance of a given system. Finally, this article proposes a set of definitions and relevant terminology to guide a program of theoretical and empirical research in this area.
PERFORMANCE MEASUREMENT AND “ORGANIZATIONAL A number of organizations researchers and theorists have concerned themselves with the issue of defining and measuring the construct of “organizational effectiveness.” The construct is, clearly, central to investigations of how different structural and process variables impact the overall success of the organization, and is thus of fundamental importance to the field. The “organizational effectiveness” literature takes the single organization as the unit of analysis, while current concern is with even more complex systems, perhaps involving several distinct organizations. However, many of the problems encountered in assessing the performance or effectiveness of such complex systems appear in efforts to assess a single organization. We shall therefore review briefly the existing work on organizational effectiveness as an introduction to our larger concern. Two major viewpoints dominate the organizational effectiveness literature. The first, the “organizational goals” (OC) approach, argues that organizations are, first and foremost, purposive or goal-seeking entities. Thus, the appropriate measures of how well they are doing come from examining how well they achieve their goals. The second approach, the “system resource” (SR) approach, argues that organizations are best viewed as open systems, which transform inputs into outputs while maintaining themselves in the process of these transformations. Thus here the appropriate measure of how well they are doing comes about from examining the effectiveness of the various inputgathering, transformation, output-disposal, and maintenance activities. A number of variants can be identified within each approach.* The Organizational Goals Approach In the “OG” approach the critical
question is, of course, how does one identify the organization’s goals? The simplest answer is to take some statement of goals such as that found in an organization’s charter, articles *The following two sections are aimed a&the reader the recent Iiterature on org~njzationa[ effectiveness* familiar with this literature, and who already share conceptual inadequacies, may wish to turn to our “The Nature of Effectiveness Statements.”
unfamiliar with Readers already our sense of its examination of
EFFECTIVENESS”
of incorporation, iegislative mandate, or similar document. Reliance on such statements has, however, at least three major drawbacks in the development of appropriate performance measures. First, the statements are invariably couched in very broad terms. For example, the official goal statement for a university might include “conduct undergraduate and graduate teaching and research”: for a police force, “maintain the public safety.” Such statements leave a very large gap between the stated goal and any conceivable operational measure of how well it is being achieved. Second, there may be only a very loose coupling between the stated goal and what the organization is actually trying to achieve. An extreme case of such loose coupling would be a commercial organization used as a “cover” for an espionage or criminal covert activity. Even excluding such extreme cases, it is a common observation that the “official” goals and the “operative” goals of an organization are distinctly different (Perrow, 1961; Porter, Lawler, & Hackman, 1976). Finally, such “official” goal statements may simply not be available for the system of interest, particularly if the focal system is an aggregate of several distinct organizations. For example, while “official” goal statements may well be available for the several elements (police, courts, corrections, etc.) of the criminal justice system, there may well be no such goals for the system as a whole; and it is far from clear that such a whole-system goal can be derived from merely adding together the goal statements of the several elements. A second cluster of problems arises as one attempts to move from “official” goals to “operative” goals (those actually being pursued), and then to “operational” goals (measurable outcomes reflecting actual goal achievement). Different individuals and groups within and outside the organization may (and probably do) have different goals. Whose are to be taken as the goals that “really matter”? It can be argued that the most appropriate goals are those of the individuals most able to influence what the organization actually does - the “major decision-makers” (Price, 1972), the “executive core” (Zald, 1963) or the “dominant coalition” (Pennings & Goodman, 1977). This approach begs the empirical question of whether or not such a
Performance
Measurement:
single dominant group actually exists in a given situation. Even within a single organization, there may be several powerful groups, each pursuing different goals; and the problem becomes yet more severe with “multinodal” systems, such as the criminal justice system, for the reasons noted above. McCormick (1973) argues for goal identification by means of a broad survey of the organizational membership, allowing empirical identification of different internal constituencies. He does not, however, resolve the issue of what is to be done if reasonable consensus is not found. Price (1973) is sharply critical of McCormick’s proposal on this and other grounds. Empirical studies such as Vroom (1960) and Lawrence and Lorsch (1967) suggest that strong goal consensus among senior managers of a single organization should not be assumed. In the same sense that operational goals can differ from operative goals in the translation downward in the organization, the upward translation of individual activities to goal statements can also depart from more formally stated goals. We refer to these goals, which may be thought of as caused by an aggregation effect as induced goals (Deutsch, Note 1). Here we do not have “major decision makers” modifying official goals permeating down through the organization but rather have perceived goals which differ from the official organization goals due to individual activities folding upward to objectives and in turn induced goals. For example, a court system might have a goal of swift administration of justice for habitual offenders whereas each individual district attorney might select cases that he feels he can win or obtain publicity from in order to promote his professional credentials. The unfolding of these activities upward in the organization would not reinforce or necessarily result in the stated official goals. Even assuming that it is possible to solve the problem of who in the organization is to be regarded as the appropriate source for operational goals statements, there remain a number of problems. First, it is far from clear that individuals in organizations have available statements of their own, or the organization’s, goals and objectives at the level of precision required to provide a basis for measurement. Indeed, experience with the technique of Management By Objectives (MBO) suggests that a major part of the work involved (and perhaps of the impact made) lies precisely in developing clear statements of objectives, with suitable measures and priorities assigned. Developing such statements is thus likely to be a large and costly undertaking for the researcher. Perhaps even more important, it is likely to be a rather significant intervention into the way the organization runs. In this light, developing adequate statements of goals and objectives is likely to be a hugely “reactive” measurement procedure (Webb, Campbell, Schwartz, & Sechrest, 1966). It may well be useful
Some Conceptual
Issues
37
for the organization, but it is a long way from the scientific norm of disturbing the system as little as possible by the act of taking measures. Finally, there is the problem of candor. If we must rely for the core of our effectiveness measurement procedure on the statements made by organizational members, we are subject to being deceived by them, either intentionally or not. It seems unlikely that most senior executives will confide in researchers their personal goals such as career advancement, prestige deriving from organizational growth, or short-term performance maximization in prospect of an early job move. Yet such strategies are being pursued, and organizational resources being devoted to them. One might thus conclude that the organization was performing rather poorly (against the chief executive’s revealed goals) while in fact it was moving very successfully to achieve the goals (s)he is really seeking. Nor is this problem obviated entirely by an “inferred goals” approach, in which the researcher infers what goals are actually being sought by observing what the organization actually does. Such observations are, of course, important. However, as an approach to defining organizational effectiveness, they imply a circular definition: goals are inferred from actual behavior; effectiveness is assessed from the comparison of actual behavior with these inferred goals. Strictly applied, all organizations would by definition show 100% effectiveness. In summary, the “organizational goals” approach attempts to specify the elements of a single set of performance measures by focusing on the organization’s goals. Official statements of such goals are likely to be vague, may not define what the organization is actually trying to do, and may not be available at all. Senior officials in the organization may be prepared to offer clearer goal statements, though this in itself is problematic, and may involve extensive research work, and extensive impact on the organization. Even if this can be done, it is not clear that senior officials will invariably agree, that their goals are the only relevant ones, or that the researcher should expect complete candor. Attempting to infer goals from actual behavior raises the danger of a circular definition of effectiveness. In short, the organizational goals approach, while offering significant insights into the assessment of organizational performance, is plagued by a number of difficulties. Systems Approaches A number of alternatives to the organizational goal approach share a linkage of some sort with systems theory, and may thus be grouped together under the broad label of “systems” approaches to effectiveness. There is, however, considerable variation between these alternatives. At one end of the spectrum is Parson’s
38
TERRY CONNOLLY
and STUART JAY DEUTSCH
functional analysis, derived from a general theory of social process. At the other are analyses such as those of Evan (1976), in which systems theory is used only to suggest categories of measurable variables which may be related to effectiveness. In between are a number of studies which offer more or less explicit linkages to some version of system theory. Some of the major formulations are discussed in the following paragraphs. In simplified terms, the basis of Parsons’ approach (Etzioni, 1960; Lyden, 1975; Parsons, 1960) is the assertion that all social systems and subsystems must solve four types of problems: they must formulate their purposes, and move toward achieving them (goal attainment); they must achieve some satisfactory adjustment to their environments (adaptation); they must coordinate their various activities (integration); and they must find a way of dealing with the strains generated in pursuing the other activities (pattern maintenance). An organization, then, is effective to the extent that it successfully solves these four problems. It should be noted that this formulation embraces the “goals” approach to effectiveness, while adding three other areas for assessment. In this sense, it is a more general view of effectiveness. As applied to organizations, open systems theory (Katz & Kahn, 1966; Yuchtman & Seashore, 1967) essentially views the organization as an entity maintaining itself while in continuous exchange with its environment. The major focus, then, is on two classes of transactions: those providing for input and output of resources; and those effecting the transformation of one into the other and the maintenance of these processes . “Resources” may include materials, money, people, energy, or information, while the “transformation process” is thought of as the entire set of technical and social mechanisms by which input resources are converted to outputs. Evan (1976) proposes a cluster of nine effectiveness measures defined in terms of the ratios between Input (I), Output (0), and Transformation (T) and their changes over time. He also suggests how these measures could be applied to a range of different sorts of organizations (such as hospitals, universities, courts and prisons). Thus for a court, the annual budget could be treated as an input, the number of cases disposed of would be treated as an output, and the cost of the supporting information system could be treated as a transformation variable. The approach, while clearly related to the “accounting ratios” approach to assessing effectiveness, does offer a more general view of effectiveness, and allows, at least conceptually, comparison of one sort of organization with another. On the other hand, it is cumbersome to apply-in any real setting, where large numbers of different inputs, transformations and outputs would be found, and where one would be faced with either attempting to reduce them all to comparable units, or dealing with
very large numbers of ratios. Neither seems completely satisfactory. A way out of this problem is offered by Yuchtman and Seashore (1967) in what they call the “systems resource” approach. They extend the argument of Evan and others by noting the further systems property of interconnectednqs. To function effectively, an organization must acquire resources, transform them, and dispose of them - and all three processes are interconnected. If, for example, transformation is done ineffectively, less valuable outputs will be produced (say, more scrap and less finished product). Therefore, fewer revenues will be generated and in turn less inputs can be acquired. Input, transformation and output are connected together in a closed loop over time, so that the effectiveness of the whole process is governed by the effectiveness of any part. In this sense, the long-term effectiveness of any of the elements of the inputtransformation-output-input loop will provide an adequate measure of the effectiveness of the whole system. Yuchtman and Seashore choose to focus on the input process: We propose . . . to define the effectiveness of an organization in terms of its bargaining position, as reflected in the ability of the organization, in either absolute or relative terms, to exploit its environment in the acquisition of scarce and valued resources. @. 898) This ingenious approach does seem to subsume several of the ideas of effectiveness noted earlier. It focuses directly on the organization’s adaptation to its environment in input terms, and indirectly addresses adaptation in output terms - a firm operating poorly in the market for its finished products will, presumably, earn reduced profits, and thus be less able to acquire new raw materials. It embraces such aspects of effectiveness as the use of human resources, both by the mechanism of reduced production, and thus profit, and by being less able to acquire new employees, or retain the ones it has. The “systems resource” approach has two major drawbacks for our current purposes. First, it relies heavily on the assumption that competitive processes are operating in the environment - that resources are acquired and disposed of in competition with other organizations. This seems a poor assumption to make of many public organizations. For example, the elements of the criminal justice system (police, courts and corrections) essentially hold monopolies of the relevant output services, and resources are acquired not in the market but by political processes. The second limitation is that the input-transformation-output loop is closed only in a long-term sense, and measures over fairly short time periods might give highly misleading indications of how effectively the system is performing.
Performance Measurement: Some Conceptual Issues Both limitations can be seen if one considers an effectiveness measure for a police department. It might, for some considerable period of time, be able to increase its budget allocation (resource acquisition) by showing that the crime rate was increasing. Since there is, generally, no rival supplier of police services in any given city, it is quite possible that the rising crime rate was the result of poor use of police resources, but that the police commissioner was able nevertheless to secure increased budgets. Ultimately, one supposes, the process would be stopped by more careful scrutiny of the police budget, and demands for improved functioning. However, for lengthy periods, it seems entirely possible that a poorly-run police department would be able to acquire growing resources, raising serious questions for approach to effectiveness the “system resource” measurement as a short-term measure of effectiveness in public systems. The point is still more sharply made if one considers a police force which devotes a large part of its resources to political lobbying. It could well acquire steadily growing resources, but would not fit our common-sense notion of an effective police organization. Even in this brief and partial review, it is clear that neither the organizational goals nor the systems approach has yet provided the last word on the issue of organizational effectiveness. Reviewers of this literature commonly offer gloomy conclusions. Thus, Steers (1975) reports “there is only a rudimentary understanding of what is actually involved in or constitutes the concept [of organizational effectiveness]“; Hrebiniak (1978) reports, “Measuring effectiveness is a critical but problematic issue in the study of organizations.“; and Campbell et al. (1974) reports, “Organizational effectiveness as it has been defined and measured in the literature is an extremely untidy construct.” An empirical comparison of the goal and system resource approach (Molnar & Rogers, 1976) found essentially no convergence between the two. Further, no quick improvement in the situation is expected. Kahn (1977) concludes that “the prospect of developing a formal theory or set of propositions (on organizational effectiveness) is still far away” (p. 248); and Steers (1975) concludes that “much remains to be done before the effectiveness construct can be usefully employed” (p. 546). Hannan and Freeman (1977), even more negatively, argue that the conceptual and methodological confusions are so deep and widespread that the concept should be abandoned altogether by serious researchers, who should instead focus on the determinants of clearly-defined, less macroscopic organizational variables. Our own conclusion is rather less pessimistic than those of these reviewers. As we shall argue in the next section, much of the pessimism in the effectiveness literature flows from an expectation that it should be
39
possible to arrive at a single statement of effectiveness, an objective and agreed-upon set of criteria by which effectiveness is to be assessed. Once this assumption is relaxed, the diversity of measures proposed in the literature becomes not an embarrassing failure of convergence, but a rich source of candidate measures. This perspective is based on a careful examination of the nature of effectiveness statements, a matter to which we now turn. The Nature of Effectiveness
Statements
A performance measure or effectiveness statement is an answer to the question: “How well is the system performing?” Such a statement might thus be of the form “System X is highly effective” or “System X has an effectiveness rating of y.” In order for such statements to have any precise meaning, the following issues must be carefully addressed: What is the Referent? That is, exactly what is being referred to as “System X “? This is almost never a trivial problem (see, for example, the discussion of system definition in Katz and Kahn [1966], or Churchman [ 1968]), and may be extremely difficult in complex social systems. For example, in defining the criminal justice system, should one include private security forces, private attorneys, non-police agencies of the federal government, and so on? Within what geographical area? Are private citizens to be treated as system members (in their roles as jurors, witnesses, complainants, etc.) or as elements of the system environment? The system definition issue is, clearly, immensely complex; but statements about how well a system is performing remain vague unless such a definition is given. What are the Data? What aspects of the system is the speaker considering in making the statement? And how adequate are the data that are being considered? For any given system, there are a potentially infinite, and practically very large number of different variables on which to base an effectiveness statement. For each, the data may vary widely on such dimensions as reliability, validity, precision, timeliness, level of measurement, and so on. An adequate effectiveness statement requires both a clear statement of the variables considered, and an assessment of the adequacy of the data used in estimating the value of each. What Transformation is Applied? An effectiveness statement is not merely an assertion of fact; it includes the application of a value scheme of some sort, the transformation of certain facts in light of certain criteria or values. Thus two individuals, given the same factual information about some specified system, might arrive at quite different assessments of the effectiveness of its performance by considering different
40
TERRY CONNOLLY
and STUART
variables as relevant, applying different weights to those considered, and by making different comparisons as to what is an adequate or desirable level for each. What is the Form of the Effectiveness Rating? Even if the preceding three issues have been satisfactorily resolved, several different forms of effectiveness statement may still be made. For example, “System X has an adequate effectiveness” implies a global rating against some external standard of “adequacy,” while “System X has an effectiveness rating of y percent” implies a global rating against a rationally-scaled external standard. The statement may imply an overall ranking or comparison, either against some other system (“System X is more effective, or y% more effective, than System Xi”), or against the previous performance of the same system (“System X is more effective, or_@70more effective, than it was last year”). Finally, the statement may distinguish between several areas of effectiveness: “System X is highly effective in Area A, moderately effective in Area B, and quite ineffective in Area CL” Such statements imply a multidimensional view of “system effectiveness.” The several dimensions may be related in various ways. For example, improvement in one area may imply deterioration in another, as in the assignment of constrained resources across areas; or performance in the various areas may be connected in a causal sequence over time, so that improvement in one leads, over time, to improvement (or deterioration) in another. En short, a statement about effectiveness is meaningful only if it is clear (1) exactly what entity is being referred to; (2) what factual information is being considered; (3) what criteria are being applied, and what comparisons made; and (4) whether the speaker considers “effectiveness” as uni- or multidimensional. In the “organizational effectiveness” literature reviewed earlier, it is apparent that the major disagreement is in the third of these areas, the criteria to be applied in transforming observational data into effectiveness ratings. The ‘*goals” approach derives its criteria from one model of the organization, the “systems” approach from another. Since the resulting sets of criteria differ, attention is directed to different sets of data, and/or to different interpretations of the same data, Not surprisingly, the resulting effectiveness statements differ. It should be emphasized that this “criterion problem” is not resolvable by examination, however detailed, of any given system. The point may be illustrated in terms of a system vastly simpler than either the single organization or the complex social system (e.g., the criminal justice system): the familiar furnace-thermostat system used in domestic heating. At first glance, it appears obvious that this system has an unambiguous purpose - the maintenance of internal temperature within certain preset limits. However, this purpose is not derivable from merely observing how the system works. Such observational data are
JAY DEUTSCH
equally interpretable in terms of a system’s purpose like “maximize fuel consumption, subject to not exceeding an upper temperature limit,” or “minimize fuel consumption, subject to not falling below a lower temperature limit.” Indeed, if the system were operated by a human thermostat and a human furnace operator, these two statements might well describe what they saw as the system purpose. This simple example illustrates two important points. First, goals and constraints are, in general, interchangeable (see also Simon [1961]; Eilon [19711). SKond, statements of purpose made by system members are likely to differ from one to another, and do not provide an unambiguous statement of “the’” system purpose. Neither is ‘Yhe” system purpose derivable from observation of system behavior. This perplexing conclusion, though trivial in the case of the thermostatfurnace system discussed above, takes on central significance when one moves to complex social systems, What are ~Terformance Measures”? So far, we have not attempted to define the term “per-
formance measure’* carefully, beyond noting that it has the flavor of a measure of how we/! a system isperforming, not just how. We have also suggested that we will generally be interested in multiple performance measures reflecting different aspects of the system’s performance, that is, in multidimensional rather than unidimensional measures. 3ut we have not yet said what we consider an adequate formal definition of the term “performance measure.” In fact, we are not able to do so in any general sense. One part of the definitional problem is that there are multiple constituencies who feel competent to comment on how well some particular system is operating, and that each is likely to feel that different aspects of the system’s performance are relevant to their assessments. For example, we are used to the idea that net profit is a useful performance measure for a business organization, indeed, perhaps the only important performance measure. However, it seems unlikely that this measure alone serves even the one-boss business. Surely the boss will also want to at least consider other measures such as growth, stability, and employee morale in assessing how well his or her system is operating, as well as measures reflecting performance over time, The complexity grows when we consider larger, more complex businesses where issues like market share and vulnerability to competitors may also be worked into the evaluation. While each of these could be argued as reievant to consideration of long-term profitability, the fact remains that the president of such a company will seek information other than current net profit in attempting to answer the question: “How well is this system operating?” A second level of complexity is introduced when we acknowledge that the president of the company may not be the oniy person who wishes to answer the question.
Performance Measurement: Some Conceptual Issues The shareholders may not entirely share the president’s view of what information is relevant (or what levels of performance are “good enough”). Employees (and their unions) may wish to have matters of job satisfaction, safety, and benefits considered in their assessments and, in many cases, will be able to force the company to pay attention to their assessments of system performance. Various agencies of government (Internal Revenue, Securities and Exchange Commission, Environmental Protection, Equal Employment Commission, and a host of others) will similarly have their own definitions of how well the system is performing, and may be in a position to do something about them. Suppliers and customers, iocal communities, the Sierra Club, and an endless list of other groups may each have legitimate interests in parts of the system’s operations. To define the measures sought by any one of these constituencies as “the” proper measure of system performance seems essentially arbitrary. We are left then, with both multiplicity (in the sense of multiple groupings with legitimate interest in the assessment of a system’s performance) and multidimensionality (in the sense of each requiring measures of several or many aspects of performance). It is clear that these factors will operate at least as sharply when we consider large public systems, such as the Criminal Justice System. The essence of the approach proposed here to the performance/effectiveness measurement problem is captured in the following three definitions: 1. A System-Relevant Constituency is an individual or group which wishes to make an assessment of how the focal system, or some part thereof, is performing, generally with a view to taking some action which will impact the system. A “constituency” is thus a (potential) user of one or more performance measures. 2. A Performance Measure is any item of systemrelevant information which affects the system performance assessment of any constituency. 3. Performance Measure Set is a grouping of performance measures in terms of the constituency to which they are relevant. The view of system performance and effectiveness measurement, then, is what might be called a “conceptual minimalist” position: we assume as little as possible a priori, preferring to treat the major issues as empirical matters, to be investigated in specific settings. Thus, we explicitly reject the goal of specifying a single, general-purpose performance measure or set of performance measures, visualizing a variety of such measures, each relevant to different constituencies. We do not assume that a single “dominant coalition” exists in all settings. As an empirical matter, such a group may be identifiable in some specific setting. They would be identified as a system-relevant constituency having sufficient po\ver to enforce their view of effectiveness on
41
the system, dominating other constituencies. The existence of such a group is here treated as a matter more appropriate for empirical study than for assumption, Similarly, the notion of a “performance measure” is treated as an empirical matter: for a given constituency, system-relevant information that makes a difference to its assessment is a performance measure. If an environmental group considers a plant’s waste water purity as a factor in its assessment of the plant’s effectiveness, then that measure becomes a part of their “performance measure set.” We should not be surprised, of course, to discover that the same measure does not appear, or is weighted less heavily, in the performance measure set of the plant’s shareholders. We thus share Hrebiniak’s (1978) view that, instead of the notion of organizational effectiveness, “it may be useful to think in terms of effectivenesses” (p. 326). Some Operational Implications At one level, the analysis and proposals we have presented here carry rather modest implications for the working evaluator or performance measure designer. In essence, they imply only that we report our results with a little more precision, and a little more humility, than we presently do. No single study can claim to have evaluation, or an “exprovided a “comprehensive” haustive” set of performance measures, bearing on a given organization or system. While it may weaken our rhetoric, it will certainly strengthen our science if we report clearly that our assessments considered only certain criteria, used only certain transformations of the data and, in brief, conducted an assessment from the perspective of only a certain constituency. No more can be demanded from any single study; and, certainly, no more should be claimed. However, such candid reform of our reporting practices sows the seeds of rather larger conceptual reforms, and rather larger changes in the day-to-day work of evaluation. if a single study is now clearly seen as reflecting the perspective of only one constituency, what other constituencies are involved, and what might we learn from applying their perspective? If, as might well be argued, current evaluation practice tends heavily to serve current power-holders, might not the technology be applied as effectively in the service of alternative constituencies - the currently dispossessed, “the public interest” (variously operationalized), employee unions, or whomever? An analysis somewhat paralleling that offered here has led at least one commentator (Keeley, 1978) to place the assessment of effectiveness squarely in the context of fundamental notions of social justice, explicitly treating the multiplicity of evaluations in terms of the equity implied for the several constituencies involved. While the operational implications flowing from this analysis are certainly profound, we do not see them as leading us to chaos. Rather, we would argue that the obsession with finding a single set of evaluative criteria
42
TERRY
CONNOLLY
and STUART
or performance measures has distorted and concealed the underlying realities. Complex social systems reflect the interplay of multiple constituencies, pursuing various purposes in the light of information of uneven quality. To design performance measures implying a simpler reality - a neat hierarchical decomposition based on clear central goals or overall system purposes - is to ensure a mismatch between information provision and decision-making realities. (See Connolly [1977], for a discussion of this matching problem in multiple-constituency decision processes.) We do not intend to comment here on methodology for identifying sets of constituencies relevant to any particular setting, nor to suggest in any detail how the relevant sets of performance measures might be developed. The identification issue leads us too deeply into the analysis of complex decisional processes; and strategies for designing measures are largely implicit in
JAY DEUTSCH
our earlier review of existing practice. Considerations of efficiency certainly suggest that effort could usefully be devoted to identifying and refining those measures - such as the various indices used in assessing the performance of a national economy - that serve the needs of several different constituencies. However, such partial intersections of different performance measure sets should not blind us to the fact that they are partial: each constituency will likely have important unique elements in its performance measure set. Even the most ambitious study cannot hope to identify all constituencies, and all performance measures, relevant to any particular system. We must choose, and must make explicit the grounds for our choice, which constituencies we are serving in any given study. In doing so, we can hope to be both clearer about just what we are doing, and more open to the complementary efforts of others developing alternative measure sets in the service of other constituencies.
SUMMARY This article has attempted to clarify a number of conceptual issues relating to the assessment of the quality of a system’s functioning, that is, the issue of performance measurement for complex systems. We first reviewed some of the extensive literature on the topic of organizational effectiveness, which considers the assessment of the performance of systems as complex as a single organization. This literature is dominated by two main approaches, one treating the organization as primarily a purposive, goal-seeking entity, the other treating it as an open system. Both between the two approaches, and within each, there is considerabfe variation in methods of arriving at an overall assessment of effectiveness. Each of the approaches reviewed seems to have important problems of one sort or another, and little convergence between the approaches was found. The literature does, however, suggest a wide range of possible performance measures. An examination of the nature of effectiveness statements suggests that such statements require clarity in at least four key areas: system definition; data used; criteria applied; and form of assessment generated, Lack of clarity in any of these areas makes the resulting statement uninterpretable. Lack of agreement in any of the areas makes two such statements incomparable. It REFERENCE 1.
DEUTSCH,
S. .l. A conceptual
basis for effectiveness
measurement
was further shown that such incomparability is most likely to arise in the area of criterion selection, and that no unambiguous determination of appropriate criteria for system performance (i.e., of “system purpose”) is to be expected from direct examination of any specific system. The proposed reso1ution of these difficulties is described as a “conceptual minimalist” approach, making as few assumptions as possible, and identifying as many issues as possible for empirical examination. The only assumptions proposed are (a) that any given compfex social system is likely to be evaluated by more than one individual or group (“system-relevant constituencies”) who can potentially affect system performance; and (b) that each such constituency will utilize some system-relevant information in making their assessment of the system’s effectiveness. The term “performance measure set” is proposed for that set of measures which can be shown to affect a given constituency’s assessment. Issues of the content of the various sets, their degree of overlap, the existence of a dominant coalition able to impose a single definition of effectiveness, and other important matters, are proposed for empirical, rather than conceptual, resolution.
NOTE of law enforcement
In Performance ~effs~fe~e~t and the D.C.: U.S. Department of Justice, 1976.
activities.
criminof justice system: Four conceptual upprouc~es (Report No. NCJ-36425). Washington,
Performance
Measurement:
Some Conceptual
Issues
43
REFERENCES CAMPELL, J. P. et al. The measurement of organizalional effecfiveness (mimeo). Minneapolis, Minn.: Personnel Decisions, 1974.
MCCORMICK, L.C. Comment on Price’s “The study of organizational effectiveness, Sociology Quarferly, 1973, 14, 271-273.
CHURCHMAN, 1968.
MOLNAR, J. J., & ROGERS, D. L. Organizational effectiveness: An empirical comparison of the goal and system resource approaches. Sociological Quarferly. 1976, I7, 401-413.
C. W. The sysfems
approach.
New York: Dell,
CONNOLLY, T. A note on the policy-relevance of evaluation. In H. W. Melton & D. J. H.Watson (Eds.), Inrerdiscipfinarydimensions of accounting for social goals and social organizations. Columbus, Oh. : Grid, 1977.
PARSONS, T. Struclure and process Free Press, 1960.
EILON, S. Goals and constraints. Journal of Management October 1971, pp. 292-303.
PENNINGS, J. M., & GOODMAN, P. S. Toward a workable framework. In P. S. Goodman & J. M. Pennings (Eds.), New perspectives on organizational effectiveness. San Francisco: JosseyBass, 1977.
ETZIONI,
A. Two approaches
ministrative
Science Quarterly,
to organizational
Studies,
in modern
PERROW, American
EVAN, W. M. Organization theory and organizational effectiveness: An exploratory analysis. Organizationaland Administrative Science, 1976, 7, 15-28.
C. The analysis of goals in complex organizations. Review, 1961, 26, 854-866.
Sociological
PORTER, L. W., LAWLER, E. E., & HACKMAN, J. R. Behavior in organizations. New York: McGraw-Hill, 1975.
GOODMAN, P. S., & PENNINGS, J. M. (Eds.). Newperspectives on organizational effectiveness. San Francisco: Jossey-Bass, 1977.
PRICE, J. L. The study of organizational
HANNAN, M. T., & FREEMAN, J. The population ecology of organizations. American Journal of Sociology, 1977, 8.7, 929-964.
PRICE,
HREBINIAK, L. G. Complexorganizations.
SIMON, H. A. Administrative millan, 1961.
New York: West, 1978.
KAHN, R. L. Organizational effectiveness: An overview. In P. S. Goodman & J. M. Pennings (Eds.), New perspectives on organizational effectiveness. San Francisco: Jossey-Bass, 1977. of organizations.
New York: Wiley, 1966.
KEELEY, M. A social-justice approach to organizational evaluation. Science Quarterly, 1978, 23, 272-292.
Administrative
LAWRENCE, P. R., & LORSCH, J. W. Differentiation and integration in complex organizations. Adminisrrafive Science Quarrerly, 1967,12, l-47. LYDEN, F. J. Using Parsons’ functional analysis in the study of public organizations. Administrative Science Quarterly, 1975, 20, 59-70.
Glencoe:
analysis. Ad-
1960, 5, 258-278.
KATZ, D., & KAHN, R. L. Thesocic....psychology
society.
Quarterly,
Quarterly,
effectiveness.
Sociology
1972, 13, 3-15.
J. L. Reply to McCormick
and Benson.
Sociological
1973, 14, 276-278. behavior
(2nd ed.). New York: Mac-
STEERS, R. M. Problems in the measurement of organizational etfectiveness. Administrative Science Quarterly, 1975, 20, 546-558. VROOM, V. H. The effects of attitudes on perceptions of organizational goals. Human Relations, 1960, 13, 229-240. WEBB, E. J., CAMPBELL, D. T., SCHWARTZ, R. D., & SECHREST, L. Unobrrusive measures. Chicago: Rand McNally, 1966. YUCHTMAN, E., &SEASHORE, S. E. A system resource approach to organizational effectiveness. American SociologicalReview, 1967, 32, 891-903. ZALD, M. M. Comparative analysis and measurement of organizational goals. Sociological Quarterly, 1963, 4, 206-230.
0147-7189/80/010035-09$02.00/O Copyright 0 1980PergamonPress Ltd
Printedin the U.S.A. Ail rightsreserved.
PERFORMANCE
MEASUREMENT:
SOME CONCEPTUAL
ISSUES
TERRY CONNOLLY and STUART JAY DEUTSCH Georgia Institute of Technology
ABSTRACT Measures of how well a system is operating are clearly of interest to a wide range of users, from organizational researchers to system managers. The current literatures in such areas as ‘performance measurement, ” “organizational effectiveness, ” and the like show little consistency in their definitions of terms or in their methods for generating measures. The present paper attempts a clarification of these issues in the form of a conceptual minimalist position which requires only three basic definitions, and leaves, asfar as possible, all remaining issues open to empirical investigation. We first review the literature on “organizational effectiveness” contrasting the organizational goals and systems paradigms, and note the lack of either theoretical or empirical convergence between the two. An examination of the nature of effectiveness statements suggests that this failure of convergence flows mainly from the different criterion sets generated by the two paradigms - and, importantly, that one should not expect convergence on a single measure or set of measures which uniquely define how well a.system is performing. One’s view of how wella given system isperforming is a function of where onestands (either theoretically or in relationship to the system), andpursuit of the one true set of performance measures is a futile exercise. Instead, we propose to redirect attention to the identification of the various individuals and groups (‘constituencies”) with an interest in system performance, and to the investigation of those items of system relevant information (their “performance measure sets”‘) which do, in fact, change their evaluations of how well the system is performing. This perspective will, we hope, redirect effort from futile theoretical debate to empirical investigations of what measures are used, by whom, and to what effect, in specific settings.
INTRODUCTION cern in this article is with performance measures in this more restrictive sense, as indicators of ho% well an organization or system is functioning, that is, system effectiveness. Such indicators are, clearly, of interest to a wide range of potential users - managers of complex systems, researchers, innovators who attempt to bring about change in system functioning, and others. This diversity of users is reflected in the diversity of definitions of performance measures, and of methods of generating them. Researchers concerned with system performance frequently measure very different things,
There appears to be no satisfactory general definition for the term “performance measure” as applied to a complex system such as an educational system, an economic system, or a criminal justice system. In the broad sense, the term could be applied to any measure which reflects some aspect of system activity - that is, to any measure which addresses the question: “How is the system performing?” The term is more frequently used in a narrower sense to denote a measure of the success, effectiveness, or desirability of the system’s behavior - that is, a measure which addresses the question: “How well is the system performing?” Our con-
This work was performed under grant number 78-NI-AX-0003 from the National Institute of Law Enforcement and Criminal Justice. Points of view or opinions stated in this document are those of the authors and do not necessarily represent the official positions or policies of the U.S. Department of Justice Requests for reprints should be sent to Terry Connolly, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332.
35
TERRY CONNOLLY
and STUART J.4Y DEUTSCH
and in very different ways.. Comparison of their findings is thus often hazardous, if not downright impossible. This article attempts to bring some conceptual clarification to the area. The first part of the article reviews the several definitions and approaches to performance measurement which have been offered by researchers concerned with the assessment of the effec-
tiveness or performance of one class of complex systems, the single organization. The second part of the article considers more closely just what is involved in making a statement about the effectiveness or performance of a given system. Finally, this article proposes a set of definitions and relevant terminology to guide a program of theoretical and empirical research in this area.
PERFORMANCE MEASUREMENT AND “ORGANIZATIONAL A number of organizations researchers and theorists have concerned themselves with the issue of defining and measuring the construct of “organizational effectiveness.” The construct is, clearly, central to investigations of how different structural and process variables impact the overall success of the organization, and is thus of fundamental importance to the field. The “organizational effectiveness” literature takes the single organization as the unit of analysis, while current concern is with even more complex systems, perhaps involving several distinct organizations. However, many of the problems encountered in assessing the performance or effectiveness of such complex systems appear in efforts to assess a single organization. We shall therefore review briefly the existing work on organizational effectiveness as an introduction to our larger concern. Two major viewpoints dominate the organizational effectiveness literature. The first, the “organizational goals” (OC) approach, argues that organizations are, first and foremost, purposive or goal-seeking entities. Thus, the appropriate measures of how well they are doing come from examining how well they achieve their goals. The second approach, the “system resource” (SR) approach, argues that organizations are best viewed as open systems, which transform inputs into outputs while maintaining themselves in the process of these transformations. Thus here the appropriate measure of how well they are doing comes about from examining the effectiveness of the various inputgathering, transformation, output-disposal, and maintenance activities. A number of variants can be identified within each approach.* The Organizational Goals Approach In the “OG” approach the critical
question is, of course, how does one identify the organization’s goals? The simplest answer is to take some statement of goals such as that found in an organization’s charter, articles *The following two sections are aimed a&the reader the recent Iiterature on org~njzationa[ effectiveness* familiar with this literature, and who already share conceptual inadequacies, may wish to turn to our “The Nature of Effectiveness Statements.”
unfamiliar with Readers already our sense of its examination of
EFFECTIVENESS”
of incorporation, iegislative mandate, or similar document. Reliance on such statements has, however, at least three major drawbacks in the development of appropriate performance measures. First, the statements are invariably couched in very broad terms. For example, the official goal statement for a university might include “conduct undergraduate and graduate teaching and research”: for a police force, “maintain the public safety.” Such statements leave a very large gap between the stated goal and any conceivable operational measure of how well it is being achieved. Second, there may be only a very loose coupling between the stated goal and what the organization is actually trying to achieve. An extreme case of such loose coupling would be a commercial organization used as a “cover” for an espionage or criminal covert activity. Even excluding such extreme cases, it is a common observation that the “official” goals and the “operative” goals of an organization are distinctly different (Perrow, 1961; Porter, Lawler, & Hackman, 1976). Finally, such “official” goal statements may simply not be available for the system of interest, particularly if the focal system is an aggregate of several distinct organizations. For example, while “official” goal statements may well be available for the several elements (police, courts, corrections, etc.) of the criminal justice system, there may well be no such goals for the system as a whole; and it is far from clear that such a whole-system goal can be derived from merely adding together the goal statements of the several elements. A second cluster of problems arises as one attempts to move from “official” goals to “operative” goals (those actually being pursued), and then to “operational” goals (measurable outcomes reflecting actual goal achievement). Different individuals and groups within and outside the organization may (and probably do) have different goals. Whose are to be taken as the goals that “really matter”? It can be argued that the most appropriate goals are those of the individuals most able to influence what the organization actually does - the “major decision-makers” (Price, 1972), the “executive core” (Zald, 1963) or the “dominant coalition” (Pennings & Goodman, 1977). This approach begs the empirical question of whether or not such a
Performance
Measurement:
single dominant group actually exists in a given situation. Even within a single organization, there may be several powerful groups, each pursuing different goals; and the problem becomes yet more severe with “multinodal” systems, such as the criminal justice system, for the reasons noted above. McCormick (1973) argues for goal identification by means of a broad survey of the organizational membership, allowing empirical identification of different internal constituencies. He does not, however, resolve the issue of what is to be done if reasonable consensus is not found. Price (1973) is sharply critical of McCormick’s proposal on this and other grounds. Empirical studies such as Vroom (1960) and Lawrence and Lorsch (1967) suggest that strong goal consensus among senior managers of a single organization should not be assumed. In the same sense that operational goals can differ from operative goals in the translation downward in the organization, the upward translation of individual activities to goal statements can also depart from more formally stated goals. We refer to these goals, which may be thought of as caused by an aggregation effect as induced goals (Deutsch, Note 1). Here we do not have “major decision makers” modifying official goals permeating down through the organization but rather have perceived goals which differ from the official organization goals due to individual activities folding upward to objectives and in turn induced goals. For example, a court system might have a goal of swift administration of justice for habitual offenders whereas each individual district attorney might select cases that he feels he can win or obtain publicity from in order to promote his professional credentials. The unfolding of these activities upward in the organization would not reinforce or necessarily result in the stated official goals. Even assuming that it is possible to solve the problem of who in the organization is to be regarded as the appropriate source for operational goals statements, there remain a number of problems. First, it is far from clear that individuals in organizations have available statements of their own, or the organization’s, goals and objectives at the level of precision required to provide a basis for measurement. Indeed, experience with the technique of Management By Objectives (MBO) suggests that a major part of the work involved (and perhaps of the impact made) lies precisely in developing clear statements of objectives, with suitable measures and priorities assigned. Developing such statements is thus likely to be a large and costly undertaking for the researcher. Perhaps even more important, it is likely to be a rather significant intervention into the way the organization runs. In this light, developing adequate statements of goals and objectives is likely to be a hugely “reactive” measurement procedure (Webb, Campbell, Schwartz, & Sechrest, 1966). It may well be useful
Some Conceptual
Issues
37
for the organization, but it is a long way from the scientific norm of disturbing the system as little as possible by the act of taking measures. Finally, there is the problem of candor. If we must rely for the core of our effectiveness measurement procedure on the statements made by organizational members, we are subject to being deceived by them, either intentionally or not. It seems unlikely that most senior executives will confide in researchers their personal goals such as career advancement, prestige deriving from organizational growth, or short-term performance maximization in prospect of an early job move. Yet such strategies are being pursued, and organizational resources being devoted to them. One might thus conclude that the organization was performing rather poorly (against the chief executive’s revealed goals) while in fact it was moving very successfully to achieve the goals (s)he is really seeking. Nor is this problem obviated entirely by an “inferred goals” approach, in which the researcher infers what goals are actually being sought by observing what the organization actually does. Such observations are, of course, important. However, as an approach to defining organizational effectiveness, they imply a circular definition: goals are inferred from actual behavior; effectiveness is assessed from the comparison of actual behavior with these inferred goals. Strictly applied, all organizations would by definition show 100% effectiveness. In summary, the “organizational goals” approach attempts to specify the elements of a single set of performance measures by focusing on the organization’s goals. Official statements of such goals are likely to be vague, may not define what the organization is actually trying to do, and may not be available at all. Senior officials in the organization may be prepared to offer clearer goal statements, though this in itself is problematic, and may involve extensive research work, and extensive impact on the organization. Even if this can be done, it is not clear that senior officials will invariably agree, that their goals are the only relevant ones, or that the researcher should expect complete candor. Attempting to infer goals from actual behavior raises the danger of a circular definition of effectiveness. In short, the organizational goals approach, while offering significant insights into the assessment of organizational performance, is plagued by a number of difficulties. Systems Approaches A number of alternatives to the organizational goal approach share a linkage of some sort with systems theory, and may thus be grouped together under the broad label of “systems” approaches to effectiveness. There is, however, considerable variation between these alternatives. At one end of the spectrum is Parson’s
38
TERRY CONNOLLY
and STUART JAY DEUTSCH
functional analysis, derived from a general theory of social process. At the other are analyses such as those of Evan (1976), in which systems theory is used only to suggest categories of measurable variables which may be related to effectiveness. In between are a number of studies which offer more or less explicit linkages to some version of system theory. Some of the major formulations are discussed in the following paragraphs. In simplified terms, the basis of Parsons’ approach (Etzioni, 1960; Lyden, 1975; Parsons, 1960) is the assertion that all social systems and subsystems must solve four types of problems: they must formulate their purposes, and move toward achieving them (goal attainment); they must achieve some satisfactory adjustment to their environments (adaptation); they must coordinate their various activities (integration); and they must find a way of dealing with the strains generated in pursuing the other activities (pattern maintenance). An organization, then, is effective to the extent that it successfully solves these four problems. It should be noted that this formulation embraces the “goals” approach to effectiveness, while adding three other areas for assessment. In this sense, it is a more general view of effectiveness. As applied to organizations, open systems theory (Katz & Kahn, 1966; Yuchtman & Seashore, 1967) essentially views the organization as an entity maintaining itself while in continuous exchange with its environment. The major focus, then, is on two classes of transactions: those providing for input and output of resources; and those effecting the transformation of one into the other and the maintenance of these processes . “Resources” may include materials, money, people, energy, or information, while the “transformation process” is thought of as the entire set of technical and social mechanisms by which input resources are converted to outputs. Evan (1976) proposes a cluster of nine effectiveness measures defined in terms of the ratios between Input (I), Output (0), and Transformation (T) and their changes over time. He also suggests how these measures could be applied to a range of different sorts of organizations (such as hospitals, universities, courts and prisons). Thus for a court, the annual budget could be treated as an input, the number of cases disposed of would be treated as an output, and the cost of the supporting information system could be treated as a transformation variable. The approach, while clearly related to the “accounting ratios” approach to assessing effectiveness, does offer a more general view of effectiveness, and allows, at least conceptually, comparison of one sort of organization with another. On the other hand, it is cumbersome to apply-in any real setting, where large numbers of different inputs, transformations and outputs would be found, and where one would be faced with either attempting to reduce them all to comparable units, or dealing with
very large numbers of ratios. Neither seems completely satisfactory. A way out of this problem is offered by Yuchtman and Seashore (1967) in what they call the “systems resource” approach. They extend the argument of Evan and others by noting the further systems property of interconnectednqs. To function effectively, an organization must acquire resources, transform them, and dispose of them - and all three processes are interconnected. If, for example, transformation is done ineffectively, less valuable outputs will be produced (say, more scrap and less finished product). Therefore, fewer revenues will be generated and in turn less inputs can be acquired. Input, transformation and output are connected together in a closed loop over time, so that the effectiveness of the whole process is governed by the effectiveness of any part. In this sense, the long-term effectiveness of any of the elements of the inputtransformation-output-input loop will provide an adequate measure of the effectiveness of the whole system. Yuchtman and Seashore choose to focus on the input process: We propose . . . to define the effectiveness of an organization in terms of its bargaining position, as reflected in the ability of the organization, in either absolute or relative terms, to exploit its environment in the acquisition of scarce and valued resources. @. 898) This ingenious approach does seem to subsume several of the ideas of effectiveness noted earlier. It focuses directly on the organization’s adaptation to its environment in input terms, and indirectly addresses adaptation in output terms - a firm operating poorly in the market for its finished products will, presumably, earn reduced profits, and thus be less able to acquire new raw materials. It embraces such aspects of effectiveness as the use of human resources, both by the mechanism of reduced production, and thus profit, and by being less able to acquire new employees, or retain the ones it has. The “systems resource” approach has two major drawbacks for our current purposes. First, it relies heavily on the assumption that competitive processes are operating in the environment - that resources are acquired and disposed of in competition with other organizations. This seems a poor assumption to make of many public organizations. For example, the elements of the criminal justice system (police, courts and corrections) essentially hold monopolies of the relevant output services, and resources are acquired not in the market but by political processes. The second limitation is that the input-transformation-output loop is closed only in a long-term sense, and measures over fairly short time periods might give highly misleading indications of how effectively the system is performing.
Performance Measurement: Some Conceptual Issues Both limitations can be seen if one considers an effectiveness measure for a police department. It might, for some considerable period of time, be able to increase its budget allocation (resource acquisition) by showing that the crime rate was increasing. Since there is, generally, no rival supplier of police services in any given city, it is quite possible that the rising crime rate was the result of poor use of police resources, but that the police commissioner was able nevertheless to secure increased budgets. Ultimately, one supposes, the process would be stopped by more careful scrutiny of the police budget, and demands for improved functioning. However, for lengthy periods, it seems entirely possible that a poorly-run police department would be able to acquire growing resources, raising serious questions for approach to effectiveness the “system resource” measurement as a short-term measure of effectiveness in public systems. The point is still more sharply made if one considers a police force which devotes a large part of its resources to political lobbying. It could well acquire steadily growing resources, but would not fit our common-sense notion of an effective police organization. Even in this brief and partial review, it is clear that neither the organizational goals nor the systems approach has yet provided the last word on the issue of organizational effectiveness. Reviewers of this literature commonly offer gloomy conclusions. Thus, Steers (1975) reports “there is only a rudimentary understanding of what is actually involved in or constitutes the concept [of organizational effectiveness]“; Hrebiniak (1978) reports, “Measuring effectiveness is a critical but problematic issue in the study of organizations.“; and Campbell et al. (1974) reports, “Organizational effectiveness as it has been defined and measured in the literature is an extremely untidy construct.” An empirical comparison of the goal and system resource approach (Molnar & Rogers, 1976) found essentially no convergence between the two. Further, no quick improvement in the situation is expected. Kahn (1977) concludes that “the prospect of developing a formal theory or set of propositions (on organizational effectiveness) is still far away” (p. 248); and Steers (1975) concludes that “much remains to be done before the effectiveness construct can be usefully employed” (p. 546). Hannan and Freeman (1977), even more negatively, argue that the conceptual and methodological confusions are so deep and widespread that the concept should be abandoned altogether by serious researchers, who should instead focus on the determinants of clearly-defined, less macroscopic organizational variables. Our own conclusion is rather less pessimistic than those of these reviewers. As we shall argue in the next section, much of the pessimism in the effectiveness literature flows from an expectation that it should be
39
possible to arrive at a single statement of effectiveness, an objective and agreed-upon set of criteria by which effectiveness is to be assessed. Once this assumption is relaxed, the diversity of measures proposed in the literature becomes not an embarrassing failure of convergence, but a rich source of candidate measures. This perspective is based on a careful examination of the nature of effectiveness statements, a matter to which we now turn. The Nature of Effectiveness
Statements
A performance measure or effectiveness statement is an answer to the question: “How well is the system performing?” Such a statement might thus be of the form “System X is highly effective” or “System X has an effectiveness rating of y.” In order for such statements to have any precise meaning, the following issues must be carefully addressed: What is the Referent? That is, exactly what is being referred to as “System X “? This is almost never a trivial problem (see, for example, the discussion of system definition in Katz and Kahn [1966], or Churchman [ 1968]), and may be extremely difficult in complex social systems. For example, in defining the criminal justice system, should one include private security forces, private attorneys, non-police agencies of the federal government, and so on? Within what geographical area? Are private citizens to be treated as system members (in their roles as jurors, witnesses, complainants, etc.) or as elements of the system environment? The system definition issue is, clearly, immensely complex; but statements about how well a system is performing remain vague unless such a definition is given. What are the Data? What aspects of the system is the speaker considering in making the statement? And how adequate are the data that are being considered? For any given system, there are a potentially infinite, and practically very large number of different variables on which to base an effectiveness statement. For each, the data may vary widely on such dimensions as reliability, validity, precision, timeliness, level of measurement, and so on. An adequate effectiveness statement requires both a clear statement of the variables considered, and an assessment of the adequacy of the data used in estimating the value of each. What Transformation is Applied? An effectiveness statement is not merely an assertion of fact; it includes the application of a value scheme of some sort, the transformation of certain facts in light of certain criteria or values. Thus two individuals, given the same factual information about some specified system, might arrive at quite different assessments of the effectiveness of its performance by considering different
40
TERRY CONNOLLY
and STUART
variables as relevant, applying different weights to those considered, and by making different comparisons as to what is an adequate or desirable level for each. What is the Form of the Effectiveness Rating? Even if the preceding three issues have been satisfactorily resolved, several different forms of effectiveness statement may still be made. For example, “System X has an adequate effectiveness” implies a global rating against some external standard of “adequacy,” while “System X has an effectiveness rating of y percent” implies a global rating against a rationally-scaled external standard. The statement may imply an overall ranking or comparison, either against some other system (“System X is more effective, or y% more effective, than System Xi”), or against the previous performance of the same system (“System X is more effective, or_@70more effective, than it was last year”). Finally, the statement may distinguish between several areas of effectiveness: “System X is highly effective in Area A, moderately effective in Area B, and quite ineffective in Area CL” Such statements imply a multidimensional view of “system effectiveness.” The several dimensions may be related in various ways. For example, improvement in one area may imply deterioration in another, as in the assignment of constrained resources across areas; or performance in the various areas may be connected in a causal sequence over time, so that improvement in one leads, over time, to improvement (or deterioration) in another. En short, a statement about effectiveness is meaningful only if it is clear (1) exactly what entity is being referred to; (2) what factual information is being considered; (3) what criteria are being applied, and what comparisons made; and (4) whether the speaker considers “effectiveness” as uni- or multidimensional. In the “organizational effectiveness” literature reviewed earlier, it is apparent that the major disagreement is in the third of these areas, the criteria to be applied in transforming observational data into effectiveness ratings. The ‘*goals” approach derives its criteria from one model of the organization, the “systems” approach from another. Since the resulting sets of criteria differ, attention is directed to different sets of data, and/or to different interpretations of the same data, Not surprisingly, the resulting effectiveness statements differ. It should be emphasized that this “criterion problem” is not resolvable by examination, however detailed, of any given system. The point may be illustrated in terms of a system vastly simpler than either the single organization or the complex social system (e.g., the criminal justice system): the familiar furnace-thermostat system used in domestic heating. At first glance, it appears obvious that this system has an unambiguous purpose - the maintenance of internal temperature within certain preset limits. However, this purpose is not derivable from merely observing how the system works. Such observational data are
JAY DEUTSCH
equally interpretable in terms of a system’s purpose like “maximize fuel consumption, subject to not exceeding an upper temperature limit,” or “minimize fuel consumption, subject to not falling below a lower temperature limit.” Indeed, if the system were operated by a human thermostat and a human furnace operator, these two statements might well describe what they saw as the system purpose. This simple example illustrates two important points. First, goals and constraints are, in general, interchangeable (see also Simon [1961]; Eilon [19711). SKond, statements of purpose made by system members are likely to differ from one to another, and do not provide an unambiguous statement of “the’” system purpose. Neither is ‘Yhe” system purpose derivable from observation of system behavior. This perplexing conclusion, though trivial in the case of the thermostatfurnace system discussed above, takes on central significance when one moves to complex social systems, What are ~Terformance Measures”? So far, we have not attempted to define the term “per-
formance measure’* carefully, beyond noting that it has the flavor of a measure of how we/! a system isperforming, not just how. We have also suggested that we will generally be interested in multiple performance measures reflecting different aspects of the system’s performance, that is, in multidimensional rather than unidimensional measures. 3ut we have not yet said what we consider an adequate formal definition of the term “performance measure.” In fact, we are not able to do so in any general sense. One part of the definitional problem is that there are multiple constituencies who feel competent to comment on how well some particular system is operating, and that each is likely to feel that different aspects of the system’s performance are relevant to their assessments. For example, we are used to the idea that net profit is a useful performance measure for a business organization, indeed, perhaps the only important performance measure. However, it seems unlikely that this measure alone serves even the one-boss business. Surely the boss will also want to at least consider other measures such as growth, stability, and employee morale in assessing how well his or her system is operating, as well as measures reflecting performance over time, The complexity grows when we consider larger, more complex businesses where issues like market share and vulnerability to competitors may also be worked into the evaluation. While each of these could be argued as reievant to consideration of long-term profitability, the fact remains that the president of such a company will seek information other than current net profit in attempting to answer the question: “How well is this system operating?” A second level of complexity is introduced when we acknowledge that the president of the company may not be the oniy person who wishes to answer the question.
Performance Measurement: Some Conceptual Issues The shareholders may not entirely share the president’s view of what information is relevant (or what levels of performance are “good enough”). Employees (and their unions) may wish to have matters of job satisfaction, safety, and benefits considered in their assessments and, in many cases, will be able to force the company to pay attention to their assessments of system performance. Various agencies of government (Internal Revenue, Securities and Exchange Commission, Environmental Protection, Equal Employment Commission, and a host of others) will similarly have their own definitions of how well the system is performing, and may be in a position to do something about them. Suppliers and customers, iocal communities, the Sierra Club, and an endless list of other groups may each have legitimate interests in parts of the system’s operations. To define the measures sought by any one of these constituencies as “the” proper measure of system performance seems essentially arbitrary. We are left then, with both multiplicity (in the sense of multiple groupings with legitimate interest in the assessment of a system’s performance) and multidimensionality (in the sense of each requiring measures of several or many aspects of performance). It is clear that these factors will operate at least as sharply when we consider large public systems, such as the Criminal Justice System. The essence of the approach proposed here to the performance/effectiveness measurement problem is captured in the following three definitions: 1. A System-Relevant Constituency is an individual or group which wishes to make an assessment of how the focal system, or some part thereof, is performing, generally with a view to taking some action which will impact the system. A “constituency” is thus a (potential) user of one or more performance measures. 2. A Performance Measure is any item of systemrelevant information which affects the system performance assessment of any constituency. 3. Performance Measure Set is a grouping of performance measures in terms of the constituency to which they are relevant. The view of system performance and effectiveness measurement, then, is what might be called a “conceptual minimalist” position: we assume as little as possible a priori, preferring to treat the major issues as empirical matters, to be investigated in specific settings. Thus, we explicitly reject the goal of specifying a single, general-purpose performance measure or set of performance measures, visualizing a variety of such measures, each relevant to different constituencies. We do not assume that a single “dominant coalition” exists in all settings. As an empirical matter, such a group may be identifiable in some specific setting. They would be identified as a system-relevant constituency having sufficient po\ver to enforce their view of effectiveness on
41
the system, dominating other constituencies. The existence of such a group is here treated as a matter more appropriate for empirical study than for assumption, Similarly, the notion of a “performance measure” is treated as an empirical matter: for a given constituency, system-relevant information that makes a difference to its assessment is a performance measure. If an environmental group considers a plant’s waste water purity as a factor in its assessment of the plant’s effectiveness, then that measure becomes a part of their “performance measure set.” We should not be surprised, of course, to discover that the same measure does not appear, or is weighted less heavily, in the performance measure set of the plant’s shareholders. We thus share Hrebiniak’s (1978) view that, instead of the notion of organizational effectiveness, “it may be useful to think in terms of effectivenesses” (p. 326). Some Operational Implications At one level, the analysis and proposals we have presented here carry rather modest implications for the working evaluator or performance measure designer. In essence, they imply only that we report our results with a little more precision, and a little more humility, than we presently do. No single study can claim to have evaluation, or an “exprovided a “comprehensive” haustive” set of performance measures, bearing on a given organization or system. While it may weaken our rhetoric, it will certainly strengthen our science if we report clearly that our assessments considered only certain criteria, used only certain transformations of the data and, in brief, conducted an assessment from the perspective of only a certain constituency. No more can be demanded from any single study; and, certainly, no more should be claimed. However, such candid reform of our reporting practices sows the seeds of rather larger conceptual reforms, and rather larger changes in the day-to-day work of evaluation. if a single study is now clearly seen as reflecting the perspective of only one constituency, what other constituencies are involved, and what might we learn from applying their perspective? If, as might well be argued, current evaluation practice tends heavily to serve current power-holders, might not the technology be applied as effectively in the service of alternative constituencies - the currently dispossessed, “the public interest” (variously operationalized), employee unions, or whomever? An analysis somewhat paralleling that offered here has led at least one commentator (Keeley, 1978) to place the assessment of effectiveness squarely in the context of fundamental notions of social justice, explicitly treating the multiplicity of evaluations in terms of the equity implied for the several constituencies involved. While the operational implications flowing from this analysis are certainly profound, we do not see them as leading us to chaos. Rather, we would argue that the obsession with finding a single set of evaluative criteria
42
TERRY
CONNOLLY
and STUART
or performance measures has distorted and concealed the underlying realities. Complex social systems reflect the interplay of multiple constituencies, pursuing various purposes in the light of information of uneven quality. To design performance measures implying a simpler reality - a neat hierarchical decomposition based on clear central goals or overall system purposes - is to ensure a mismatch between information provision and decision-making realities. (See Connolly [1977], for a discussion of this matching problem in multiple-constituency decision processes.) We do not intend to comment here on methodology for identifying sets of constituencies relevant to any particular setting, nor to suggest in any detail how the relevant sets of performance measures might be developed. The identification issue leads us too deeply into the analysis of complex decisional processes; and strategies for designing measures are largely implicit in
JAY DEUTSCH
our earlier review of existing practice. Considerations of efficiency certainly suggest that effort could usefully be devoted to identifying and refining those measures - such as the various indices used in assessing the performance of a national economy - that serve the needs of several different constituencies. However, such partial intersections of different performance measure sets should not blind us to the fact that they are partial: each constituency will likely have important unique elements in its performance measure set. Even the most ambitious study cannot hope to identify all constituencies, and all performance measures, relevant to any particular system. We must choose, and must make explicit the grounds for our choice, which constituencies we are serving in any given study. In doing so, we can hope to be both clearer about just what we are doing, and more open to the complementary efforts of others developing alternative measure sets in the service of other constituencies.
SUMMARY This article has attempted to clarify a number of conceptual issues relating to the assessment of the quality of a system’s functioning, that is, the issue of performance measurement for complex systems. We first reviewed some of the extensive literature on the topic of organizational effectiveness, which considers the assessment of the performance of systems as complex as a single organization. This literature is dominated by two main approaches, one treating the organization as primarily a purposive, goal-seeking entity, the other treating it as an open system. Both between the two approaches, and within each, there is considerabfe variation in methods of arriving at an overall assessment of effectiveness. Each of the approaches reviewed seems to have important problems of one sort or another, and little convergence between the approaches was found. The literature does, however, suggest a wide range of possible performance measures. An examination of the nature of effectiveness statements suggests that such statements require clarity in at least four key areas: system definition; data used; criteria applied; and form of assessment generated, Lack of clarity in any of these areas makes the resulting statement uninterpretable. Lack of agreement in any of the areas makes two such statements incomparable. It REFERENCE 1.
DEUTSCH,
S. .l. A conceptual
basis for effectiveness
measurement
was further shown that such incomparability is most likely to arise in the area of criterion selection, and that no unambiguous determination of appropriate criteria for system performance (i.e., of “system purpose”) is to be expected from direct examination of any specific system. The proposed reso1ution of these difficulties is described as a “conceptual minimalist” approach, making as few assumptions as possible, and identifying as many issues as possible for empirical examination. The only assumptions proposed are (a) that any given compfex social system is likely to be evaluated by more than one individual or group (“system-relevant constituencies”) who can potentially affect system performance; and (b) that each such constituency will utilize some system-relevant information in making their assessment of the system’s effectiveness. The term “performance measure set” is proposed for that set of measures which can be shown to affect a given constituency’s assessment. Issues of the content of the various sets, their degree of overlap, the existence of a dominant coalition able to impose a single definition of effectiveness, and other important matters, are proposed for empirical, rather than conceptual, resolution.
NOTE of law enforcement
In Performance ~effs~fe~e~t and the D.C.: U.S. Department of Justice, 1976.
activities.
criminof justice system: Four conceptual upprouc~es (Report No. NCJ-36425). Washington,
Performance
Measurement:
Some Conceptual
Issues
43
REFERENCES CAMPELL, J. P. et al. The measurement of organizalional effecfiveness (mimeo). Minneapolis, Minn.: Personnel Decisions, 1974.
MCCORMICK, L.C. Comment on Price’s “The study of organizational effectiveness, Sociology Quarferly, 1973, 14, 271-273.
CHURCHMAN, 1968.
MOLNAR, J. J., & ROGERS, D. L. Organizational effectiveness: An empirical comparison of the goal and system resource approaches. Sociological Quarferly. 1976, I7, 401-413.
C. W. The sysfems
approach.
New York: Dell,
CONNOLLY, T. A note on the policy-relevance of evaluation. In H. W. Melton & D. J. H.Watson (Eds.), Inrerdiscipfinarydimensions of accounting for social goals and social organizations. Columbus, Oh. : Grid, 1977.
PARSONS, T. Struclure and process Free Press, 1960.
EILON, S. Goals and constraints. Journal of Management October 1971, pp. 292-303.
PENNINGS, J. M., & GOODMAN, P. S. Toward a workable framework. In P. S. Goodman & J. M. Pennings (Eds.), New perspectives on organizational effectiveness. San Francisco: JosseyBass, 1977.
ETZIONI,
A. Two approaches
ministrative
Science Quarterly,
to organizational
Studies,
in modern
PERROW, American
EVAN, W. M. Organization theory and organizational effectiveness: An exploratory analysis. Organizationaland Administrative Science, 1976, 7, 15-28.
C. The analysis of goals in complex organizations. Review, 1961, 26, 854-866.
Sociological
PORTER, L. W., LAWLER, E. E., & HACKMAN, J. R. Behavior in organizations. New York: McGraw-Hill, 1975.
GOODMAN, P. S., & PENNINGS, J. M. (Eds.). Newperspectives on organizational effectiveness. San Francisco: Jossey-Bass, 1977.
PRICE, J. L. The study of organizational
HANNAN, M. T., & FREEMAN, J. The population ecology of organizations. American Journal of Sociology, 1977, 8.7, 929-964.
PRICE,
HREBINIAK, L. G. Complexorganizations.
SIMON, H. A. Administrative millan, 1961.
New York: West, 1978.
KAHN, R. L. Organizational effectiveness: An overview. In P. S. Goodman & J. M. Pennings (Eds.), New perspectives on organizational effectiveness. San Francisco: Jossey-Bass, 1977. of organizations.
New York: Wiley, 1966.
KEELEY, M. A social-justice approach to organizational evaluation. Science Quarterly, 1978, 23, 272-292.
Administrative
LAWRENCE, P. R., & LORSCH, J. W. Differentiation and integration in complex organizations. Adminisrrafive Science Quarrerly, 1967,12, l-47. LYDEN, F. J. Using Parsons’ functional analysis in the study of public organizations. Administrative Science Quarterly, 1975, 20, 59-70.
Glencoe:
analysis. Ad-
1960, 5, 258-278.
KATZ, D., & KAHN, R. L. Thesocic....psychology
society.
Quarterly,
Quarterly,
effectiveness.
Sociology
1972, 13, 3-15.
J. L. Reply to McCormick
and Benson.
Sociological
1973, 14, 276-278. behavior
(2nd ed.). New York: Mac-
STEERS, R. M. Problems in the measurement of organizational etfectiveness. Administrative Science Quarterly, 1975, 20, 546-558. VROOM, V. H. The effects of attitudes on perceptions of organizational goals. Human Relations, 1960, 13, 229-240. WEBB, E. J., CAMPBELL, D. T., SCHWARTZ, R. D., & SECHREST, L. Unobrrusive measures. Chicago: Rand McNally, 1966. YUCHTMAN, E., &SEASHORE, S. E. A system resource approach to organizational effectiveness. American SociologicalReview, 1967, 32, 891-903. ZALD, M. M. Comparative analysis and measurement of organizational goals. Sociological Quarterly, 1963, 4, 206-230.