- Email: [email protected]
A framework to assess the development and capacity of non-profit agencies
Evaluation and Program Planning 29 (2006) 171–179 www.elsevier.com/locate/evalprogplan
A framework to assess the development and capacity of non-profit agencies Russell G. Schuh a,*, Laura C. Leviton b a
Department of Administrative and Policy Studies, School of Education, University of Pittsburgh, 5906 Wesley W. Posvar Hall, Pittsburgh, PA 15260, USA b Research and Evaluation, Robert Wood Johnson Foundation, Route 1 North and College Road East, Princeton, NJ 08543, USA Received 1 November 2005; received in revised form 15 December 2005; accepted 30 December 2005
Abstract In this article, we present a framework to assess non-profit agencies’ levels of development and capacity. The framework is useful for evaluation because it helps to predict and explain program implementation. The framework is useful for program planning because planners must select organizations that are suitable to implement programs, and sometimes they must build organizations’ capacity to do the work. A scoring rubric called the stages of organizational change measures development and capacity, using the maturity modeling approach. Maturity modeling is a technique that emerged from the worlds of business and technology. ‘Maturity’ does not imply value judgments about organizations; rather, the level of development and capacity should fit the services to be delivered. The scoring rubric has good reliability and validity and has been applied in three ways: (1) to assess agency capacity at a single time point; (2) to describe capacity development; and (3) to evaluate capacity building initiatives. q 2006 Elsevier Ltd. All rights reserved.
1. Introduction Among all the forces affecting program implementation and survival, the organizational context has received surprisingly little formal consideration. ‘Formal consideration’ means the exploration and measurement of generic organizational features that predictably affect programs. Programs are not the same things as the agencies that house them, yet evaluation has failed to formally consider both the organization’s stage of development and the organizational characteristics that support the capacity necessary for non-profit agencies to do their work well over time. Formative or implementation evaluations inherently must address questions of organizational capacity. Programs are not some idealized models and they do not take place in Petri dishes. They are implemented largely within organizations, which have strengths and weaknesses, other activities, and other agendas. The literature on organizations and systems reveals that some of these forces are commonly understood, and others are becoming better understood. Insights about the characteristics of non-profit agencies (such as leadership and governance, organizational depth, and financial sustainability) can help us predict the quality of * Corresponding author. Tel.: C1 412 624 7659; fax: C1 412 648 1784. E-mail addresses: [email protected] (R.G. Schuh), [email protected] (L.C. Leviton).
0149-7189/$ - see front matter q 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2005.12.001
program implementation, explain outcomes, and inform the external validity of outcome conclusions. In this article, we present a framework for considering how non-profit agencies’ development and capacity affect program implementation.1 The framework has been applied to 56 nonprofit agencies to date. It has been used in three ways: (1) to assess agency capacity at a single time point; (2) to describe capacity development; and (3) to evaluate capacity building initiatives. First, we present an initial orientation to the framework and definitions of terms, beginning to relate the framework to evaluation and program planning. We then review the empirical and theoretical underpinnings of the framework, along with the method we have developed for rating organizational features according to the framework. Finally, we provide some initial insights from the framework that may assist evaluation and program planning. 2. Basic orientation and definitions Non-profit agencies differ greatly in the degree to which they have developed five key organizational features: Governance, financial resources, organizational development (i.e. structure and differentiation), internal operations, and core services. These five features work through two levels of 1
Governmental agencies are also at different stages of development in ways similar to the ones studied here. However, the framework has not been formally applied to such agencies as of this writing.
172
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
organizational capacity—individual expertise and organizational procedures and resources (see definitions below). Although programs are heterogeneous in terms of services and populations served, by contrast development of these five organizational features at two levels of capacity appear to be generic to all non-profit agencies that provide services. The constructs and theories underlying the framework are the focus of this article. They are based on precepts from diverse sources of which the most important are maturity modeling, organizational life-cycle theory, systems theory, cognitive psychology, and educational research. These five organizational features are being explored using a scoring rubric, the Stages of Organization Capacity (SOC), designed to assess agencies’ stages of development. The SOC relates theoretical constructs to observable attributes within each feature, by rating their stages of development and capacity. The framework, rather than the SOC rating instrument, is the focus of this article. Instrumentation is satisfactory at this stage, but the SOC continues to undergo revision for a very important data-based reason. The observations derived from the SOC are starting to teach us new things about the nomological net: that is, the set of underlying relationships among the features and changes in the features of the SOC. This is the sign of a useful instrument, and is predicted as one of the criteria for construct validation (Cronbach & Meehl, 1955). The nomological net provides a framework for relating observable attributes to each other. Both the framework and SOC are demonstrating their value in helping us improve our understanding of the fate of programs and services within non-profit agencies. To our knowledge, most of these dynamics have not been described in the research literature on non-profit agencies or evaluation—a significant gap in the literature about non-profit organizations and program evaluation that we are making an initial effort to fill in with this paper. 2.1. Definition: non-profit organizational capacity We have settled on the following operational definition of organizational capacity: the ability to successfully implement and complete a new project or to expand an existing one successfully. Other operational definitions are reasonable, but this one focuses on characteristics that are amenable to change and that affect non-profits’ ability to carry out services. An organization’s ‘projects’ are not limited to service delivery, but can include a fund raising drive, public relations program, and other non-program, non-service delivery efforts. When agencies are out of equilibrium (i.e. expanding or spiraling down), need quick response to volatile environments, take on new work, or expand their missions, organizational capacity and capacity building become issues. An organization is analogous to a glass of water, and its capacity is analogous— not to the size of the glass—but to how much more water the glass can carry without spilling some. The research literature to date suggests two levels of organizational capacity that, when enhanced or more fully
developed, tend to improve the likelihood of successful program implementation (Letts, Ryan, & Grossman, 1999): † Individual expertise that includes the skills, knowledge, and experience that employees and volunteers bring to the organization (in this framework ‘leadership’, is a type of expertise, to be expressed in a particular organizational context). † Organizational resources and procedures that permit agencies to use individual expertise productively. Greater organizational differentiation will assist individuals to work at a higher level, using more specialized skills. An example is the use of support staff to handle repetitive work, freeing more skilled staff to tackle the more complex tasks. Less developed agencies are unable to function at the same level of specialization. When capacity building is the goal of interventions, too often the focus is on individual training, to enhance expertise and make higher order task performance possible. Yet, the organization may not be able to use the increased expertise without some changes in its own processes and resources. This pattern is abundantly clear in the literature on medical care quality improvement, where the focus has shifted to improving the processes of medical care because individual intervention has small effects (Berwick, Godfrey, & Roessner, 1991). 2.2. Definition: non-profit organization development We define development in terms of the level, or stage, of five key features that comprise the framework. They were selected because, according to the literature, they: (a) are common to all non-profit agencies, (b) are likely to change as agencies develop, (c) can develop at different rates, and (d) affect an organization’s ability to implement new or expanded community innovations. We chose these features based on evidence from the organizational life-cycle literature suggesting that agencies have a general pattern of maturation in common, and that these features vary with organizational stage of development. Quinn and Cameron (1983) present evidence that the same patterns emerge in a wide range of settings, structures, and functions. The five features are: 1. Governance—legal status of organization and associated documents; board composition, functioning, and development; relationship between board and staff. 2. Financial resources—organization’s funding history and its ability to generate funds and to sustain itself; funding resiliency, diversity, and vulnerability. 3. Organizational development—degree of horizontal and vertical structure of the organization; administrative versus service role differentiation; skills and experience of administrative staff; role differentiation and specialization 4. Internal operations—degree to which an organization has differentiated its administrative functions, degree to which tasks are specialized by degree of skill or training they require. Often internal operations are considered overhead
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
or support operations (e.g. executive leadership, financial systems, human resource systems, and management information systems (MIS)). 5. Core services—degree to which services (or key businesses) of an organization are differentiated and the skill and experience of staff in providing them.
2.3. Relevance to evaluation and program planning 2.3.1. Program capacity versus organizational capacity The program to be evaluated is different from the agency that houses it. Thus, program capacity needs to be distinguished from organizational capacity (see Hunter, 2006; Hunter & Koopmans, 2006). The latter relates to agencies’ ability to expand or to start something new. As described below, the feature of core services relates most closely to program capacity, except that many non-profit agencies have more than one core service. Apparent problems of program capacity, such as slow start up of a program or faulty implementation of service delivery, are sometimes symptoms of organizational capacity problems. Planners and funders often believe that capacity is impaired when agencies lose a program or a funding stream. However, in using the framework we have observed quite the opposite: challenges to organizational capacity are greatest when a new program begins, not when it is ended. At the beginning, staff must be hired or diverted from other functions; new organizational procedures must be put in place, and sometimes a new core service (or core business) must be introduced. In a real sense, organizational capacity is greatest when a program is at its end. Staff is available for re-assignment, procedures have been worked out, and the program has been integrated (more or less) into the overall conduct of the non-profit agency’s work. 2.3.2. The relationship between non-profits’ development and capacity Capacity and development do not necessarily go hand in hand. Highly developed agencies that are already operating at full capacity can experience capacity problems when they add something new, just as less developed agencies can. The difference is that, if less developed agencies are to generate additional capacity, they will often need to become more formally developed. Programs are often planned and funded that require an agency to use more formal procedures and methods than their stages of development can support. For example, with a new or expanded program, the payroll might increase from five staff to 25, creating a need for more formally developed internal operations. The funder of the new program might require a more formal financial system and more formal performance reporting than the agency has in place. More formal systems require greater effort on the part of staff to complete records and follow procedures, and less developed agencies often need to grow these organizational features at the same time that they are expanding services or taking on a new
173
program, which can produce predictable stresses both on these features and on the programs. 2.3.3. Challenges to value judgments At this point in the exposition, we need to confront some implicit value judgments about organizational development and capacity. Development does not connote ‘better’ or ‘worse’ services. Rather, there are trade-offs for providing services. Both informally and more formally developed agencies have their places in service delivery. Agencies implement services differently depending, in part, on their degree of formal development. Agencies that operate more informally can have advantages for certain services: for other purposes a more developed organization will be appropriate because it can better accommodate a particular program. Some agencies should remain informal precisely because they fill an important niche and do not require the kind of development that is often a focus of funders (see for example, Leviton, Herrera, Pepper, Fishman, & Racine, 2006). In fact, there are sound reasons for some agencies to reject more formal development. Small, horizontal agencies can be more innovative than larger, more formal agencies—they have more capacity for rapid innovation. Grassroots agencies are often more sensitively responsive to their environments, as seen in the many examples of communities’ need to cope with exposures to toxic environmental agents or to serve people with AIDS. Bureaucracies could not adapt in a timely way to assist or to meet the community on its own terms—they had limited capacity to evolve and address their needs (Edelstein & Wandersman, 1987; Leviton, Needleman, & Shapiro, 1998). Some grassroots agencies can provide more effective services to community members, compared to more developed agencies. They have more capacity to provide these services because their staffs possess expertise about the community, and the organization’s procedures permit its expression. More formally developed agencies will tend to have fewer problems implementing programs under at least three conditions: when the program is new, when it is expanding (as described above); or when a pre-specified, complex program model must be implemented with fidelity. The problem of implementing with fidelity has been encountered in many programs. Recently, program planners and funders have started to look for agencies that are sufficiently developed to implement the models with fidelity (Miller, Bedney, & Guenther-Gray, 2003). The framework offers three testable hypotheses about such agencies, as seen at the end of this article. 3. Development of the framework 3.1. Maturity models to assess agencies Agencies are fluid and dynamic. From our observations, they can sometimes develop or devolve quite rapidly. Based on the literature concerning organization life-cycles, it seems important to measure stages of development. The research literature points to a useful approach to capture these
174
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
dynamics: the maturity model. A maturity model involves a set of features and a related set of levels or stages for each feature (Lesgold, 2003). The stages or levels represent a developmental continuum arranged from initial, or beginning, condition to mature, or fully developed, condition. Maturity models extend evaluators’ abilities to track change in three important ways: (a) they can be developed for any number of features, (b) they emphasize the development of scoring rubrics to facilitate reliable assignment of each feature to the most appropriate stage or level, and (c) they have already been used extensively to assess organizational-level innovations in educational and technology transfer environments (Kazanjian & Drazin, 1990). ‘Maturity model’ is a technical term. It does not imply a value judgment or any prediction about the direction that capacity or development will take. The flexibility of maturity models for assessing both individual- and organization-level attributes makes them ideal for addressing the complex issues of agency capacity. A scoring rubric is a taxonomy of observed characteristics or symptoms. The taxonomy can accrue, becoming more differentiated through development and use, and therefore more reliable and precise (Frick & Semmel, 1978). The stages or levels of the features are arranged from those one would expect to find in a less mature organization to those one would expect in the most mature organizations. Each feature has attributes for which scoring rubrics have been developed. In developing the SOC scoring rubric, an iterative process was used, consisting of (a) identification of the core set of organizational attributes from the literature that were postulated to change as an organization matured, followed by (b) retrospective application to an organization with data on file from previous work, followed by (c) revision as indicated by the retrospective application, and (d) application of the revised instrument to another case. The maturity model continued to be refined by the addition of attributes as experience was gained in the field (both calibrated and uncalibrated at the present time). To illustrate the scoring rubric, in Fig. 1 we present some examples of attributes for the feature of financial resources (the soc scoring manual is available from the first author). As seen in these examples, the staging of attributes is done on the basis of specific anchor descriptions, wherever possible. In other words, the description typifies the attribute at a specific stage of development. Uncalibrated indicators will often point to movement from one stage to another. In other words, these would provide evidence that the organization is developing more mature features, or devolving to less mature features. Such movement provides a basis for further inquiry about the agency. 3.2. Assessing individual expertise The SOC rubric uses two levels of maturity modeling: one for organizational processes and one for individual expertise. A research literature on individual expertise, derived from cognitive psychology, applies to the non-profit setting in a straightforward way. Expertise develops in stages (Ericsson & Smith, 1991; Hoffman, 1992; Simon, 1974). ‘Expertise’ is
often described only as possessing extensive training and experience in a given field. However, it is possible for individuals to possess both extensive experience and training in a field and, at the same time, fail to achieve expert performance. Expertise goes beyond experience and training, to learning to understand and solve problems in different ways, a skill that grows to be intuitive in a true expert. We adapted a five-stage model from the work of Dreyfus, Dreyfus, and Athanasiou (1986): Stage 1 Novice. Uses simple rule-based problem solving; uses ‘backward thinking’ to envision a solution and follows rules to get there. Stage 2 Advanced beginner. Uses backward thinking with a greater number of rules; situational thinking begins to emerge. Stage 3 Competent. Uses situational problem solving (different rules for different circumstances); can adopt a perspective and follow it to a solution. Stage 4 Proficient. Uses multiple perspectives, rapidly assessing which to adopt; understands relevant data; forward thinking emerges. Stage 5 Expert. Uses intuitive forward-thinking problem solving; recognizes underlying patterns. Expertise is domain-specific and requires years of practice to develop (Simonton, 1996). For example, in using the framework we have encountered professionals who were very proficient service providers, but then became inexperienced managers. As managers they started over as novices, or at best merely competent, because expertise is domain-specific. The SOC has also revealed how organizational limitations of capacity effectively reduce the expertise of highly proficient people. For example, an information technology professional was trying to develop an interagency information system. However, he was reduced to entering data and fixing network problems, lower order tasks for which he had no staff support. 3.3. Data collection and validation One or more outside observers use the scoring rubric to produce a profile of organizational stage of development. Ideally, observers have extensive experience in studying or working with agencies and their behavior, because such individuals will more readily observe underlying patterns. Developing the profile of a non-profit organization occurs in stages. Data collection for SOC is unstructured, qualitative, and iterative—that is, it goes in cycles. Quantification and structure come later, after each round of data collection, once the rater’s impressions are applied to the scoring rubric. This then becomes the basis of further inquiry to fill out a more detailed profile. Not every attribute of every scoring rubric has to be rated to gain an overall picture of development for each of the five features. The level of detail is dictated by the purpose of data collection. Reliability and validity information is accruing along with the taxonomy. To date, various versions of the scoring rubrics
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
175
Agency Stage of Development
Features Stage 1 Initial, start-up or small agency Financial Resources Maturity (sample attributes)
C
Stage 3 Established agency
Stage 4 Proficient agency
Stage 5 Highly productive agency
Unstable, unpredictable or insecure funding
Limited funding, more stable than stage 1 agency
Established agency funding where organization has some control over its own destiny
Stable sources of income or funding
Highly stable funding
Opportunistic funding (like that resulting from a new or emerging policy concern).
Contingency funding for activities or services but not long term (e.g. summer youth activities).
Initiative funding Project funding derived from services related to an on-going policy priority like violence prevention or early childhood development.
Niche funding Agency funded through on-going source like United Way or local funding authority (e.g. mental health, housing or juvenile justice).
Institutionalized funding for at least a core of services or programs (e.g. Funding is a line item on a state or local authority budget.
Start-up or seed money only or short term funding.
Projects or activities funded on a one-time basis (funding may span several years).
Funding may be for specified periods but contains provision for renewal or continuation.
Core funding ongoing, may have provision for periodic review or renewal.
Core service funding is highly stable. May be supported by spinoff services or products.
A
B
Stage 2 Advanced young or small agency
Fig. 1. Examples of scoring rubric for financial resources.
have been applied to 56 agencies. Interrater reliability (kappa statistic, not percent agreement) ranges from 0.81 to 0.86 for the features of governance, financial resources and organizational development.2 Interrater reliability (kappa) for the features of internal operations and core services are currently in the 0.6 ranges. The difference may be due, in part, to availability of literature pointing to common principles for the first three features, which allowed comparison across organizations. No such literature was available for internal operations or core services, so the attributes presented are all original work and developed ‘from scratch’. Because many relevant features and attributes are derived from the literature on non-profit agencies and organizational life-cycles, we have some confidence in the construct validity of the framework and SOC scoring rubric. Furthermore, the emerging relationships as assessed by the SOC scoring rubric provide some confidence in the utility of the framework for understanding the nomological net. Finally, face validity of the SOC scoring rubric is quite high, because executive directors and board members have all indicated that the SOC profiles reveal good understanding of the agencies’ stage of maturity. 4. The framework in detail 4.1. Governance Governance strongly affects other features of interest to program planning and evaluation. Governance includes the 2 Kappa statistics adjust for the probability of a given response and are therefore preferable to percent agreement.
board of directors and the organization’s legal relationship with the environment, including charter, type of organization, and mission. Boards of new and less developed agencies tend to be small and operate informally. As agencies mature, boards tend to grow in size and become more formal in their structure and work. Board focus also changes as agencies develop, from rules-based requirements of getting established, to stabilization of the organization’s resources, and finally to growth and expansion of mission. The relationship between boards and organization staff also changes as agencies develop. Staff leadership in early stages evolves into greater differentiation between the functions of management (by staff) and policy (by board). Development of this domain was based on literature from systems theory, particularly open systems theory, and organizational development (Hall & Fagen, 1968; Katz & Kahn, 1966). Board development can be a focus in organizational capacity building, as in a recent initiative of the Robert Wood Johnson Foundation (RWJF) evaluated by the first author (Schuh, 2005). The evaluation revealed very little change in boards due to this initiative. The reasons may be explained, in part, by certain attributes of the agencies. In general, as agencies mature, their governing boards focus increasingly on organization development and growth. However: 1. Several agencies had mandates to include consumers, public officials, or representation by other agencies. Boards with mandated representation may retain a greater interest in advocating for their constituency (a legitimate focus), than in growing the organization.
176
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
This feature addresses the organization’s ability to secure resources for sustainability and mission development. Agencies can progress from unstable to highly stable funding, although they may lose ground from shifts in the environment. Research on the non-profit sector, management, and organization development has contributed to this domain. For example, it is clear that sustainability is related, in part, to organizational performance (Letts et al., 1999; Light, 1998). The relationship to implementation should be readily apparent: if an agency cannot meet the payroll, it is unlikely the program can be well-implemented or sustained in that agency. Financial resources also affect the degree to which agencies must crosssubsidize other organizational functions using program resources. Financial maturity is an important subset of attributes under this feature. It is reflected in routinized or formalized activities to secure funds: for example, provisions for recovering agency overhead costs, yearly fund raising campaigns, or new product development. More financially mature agencies can afford to choose funding sources that will pay for activities that are in line with their mission, rather than on an opportunistic basis. Fund raising that reflects greater financial maturity is seen in an absence of crisis orientation (e.g. to meet the payroll or the rent), responsibilities shared by the staff and the board, routinized or predictable fund raising activities, and a more diverse or strategic set of fund raising activities. A second important subset concerns financial vulnerability, the ability to stay in operation despite a variety of financial stresses or agency transitions. Vulnerability is reflected in whether an agency has equity, a diversity of funding sources and types of funders (Greenlee & Trussel, 2000; Tuckman & Chang, 1991), adequate operating margins, and defined administrative costs that support the organization’s infrastructure. These are all important predictors of organization survival (see also Herrera et al., this issue).
(often called vertical and horizontal differentiation) change with development and growth. Tall organizational structures are often older and more formal than short ones, and communicate more formally as well (Downs, 1967; Quinn & Cameron, 1983). Short structures can be an indication of less developed administrative infrastructure (Hanks, Watson, Jansen, & Chandler, 1994). In general, non-profit agencies tend to under-emphasize infrastructure development—some would say they starve it because of the priority placed on providing services (Letts et al., 1999). In less developed agencies, the roles of administration and of service delivery are often undifferentiated, with agency leaders ‘wearing several hats’. Administration and service roles become differentiated as agencies develop infrastructure and capacity. Larger and more differentiated administrative infrastructures are often an indicator of highly productive and financially robust agencies. Such agencies, in turn, tend to have greater administrative specialization, because the infrastructure helps people do the work that requires their highest skill levels most of the time. For example, a more vertically differentiated organization will have specialists in human resources, budget, communications, fund raising and management information systems. By contrast, in less developed agencies, it is common to find the executive director cleaning offices, sending notices to the board, and at the same time preparing the financial report. It is also common to find executive directors providing direct services, conducting fund raisers, and serving as the human resources officer. Since expertise is highly domain specific, working in a variety of roles precludes application of one’s greatest expertise much of the time. Observations about role specialization and administrative differentiation provide insights about the implementation of programs. For example, greater administrative differentiation is reflected in greater overhead costs, so a comparable program will appear to cost more in more developed agencies than in less developed organizations. However, the savings in less developed agencies may not be real, because many of the overheadsupported functions found in more mature agencies are supported through cross-subsidization in less mature agencies. Undifferentiated agencies tend to characterize themselves as having cultures in which ‘everyone does everything’ or they may say ‘we wear many hats’. Usually only one hat is supported from the outside, or at least some hats will not be supported. This is a form of cross-subsidization. The extent of cross-subsidization limits an agency’s capacity to take on additional projects, or to implement a program with fidelity—both of which can be very significant opportunity costs.
4.3. Organization development
4.4. Internal operations
This feature addresses the manner in which agencies structure themselves to achieve their missions. It has been informed by organizational, management, and non-profit literature (Harrison, 1987; Lawrence & Lorsch, 1969; Letts et al., 1999). The height and width of the organization chart
Internal operations in non-profit agencies are devised over time, and there can be great variation in these arrangements. However, each internal operation can be compared to others in terms of the developmental maturity of their infrastructure (space, administrative components, and logistic support);
2. Some agencies were sub-units of parent organizations, and under these conditions the agency board has limits on its ability to influence direction and policy. 3. Most of the agencies suffered from ‘founder’s syndrome’ in which an organization cannot develop beyond the vision of a strong leader–founder. Since boards tend to focus more on operations as agencies mature, the presence of a strong founder hindered their transition to a focus on operations. We find references to founder’s syndrome in the literature, but no explanation of the underlying dynamics.
4.2. Financial resources
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
technology (computers, specialized equipment); role differentiation (horizontal separation of tasks by domain); role specialization (vertical separation of tasks by skill within a domain); and expertise of those providing the internal operation. This is also the approach employed to score the maturity of core services, because it accommodates great variation in organizational arrangements while permitting comparison across agencies and within agencies across time. The internal operations that are the framework’s current focus include management information systems (MIS), which can be informal, as with notes on the back of a napkin, or formal as with computerized MIS. In the same way, financial operations can be informal, as in writing checks by hand, balancing a checkbook, and issuing ad hoc reports. It can be formal, as with computerized monitoring of project expenditures and regularly generated balance sheets and income projections. The attributes of human resources, communications, and development (as in product development and fundraising) all reflect agency stage of development. The rubric can be expanded to issues such as planning, where necessary. 4.5. Core services Within core services, SOC raters score each separate project. Projects can reflect different organizational activities, but we also distinguish more than one project when different funding sources require different things from the same activity. This gets to the specifics of what the agency is implementing at the street level. For example, drug treatment may be funded through block grants from the state, or directly by Medicaid, and the requirements shape the specific interventions. Core services largely define a community agency, which is often founded with the idea of addressing a perceived need. Services, like internal operations, are devised over time and can vary infinitely. As with internal operations, certain attributes appear consistently, offering comparison across agencies and within a developing agency across time. The attributes of core services, like those of internal operations, include infrastructure, technology, role differentiation and specialization, and expertise of the key service providers. 5. Learning from the framework 5.1. Patterns of growth and diagnosis of capacity needs Experience to date leads us to observe a general pattern: core services tend to develop first, followed by some aspects of financial management. What is surprising is that development of administrative infrastructure often lags by a stage and that board development and functioning generally lags even more. From observation, it appears that significant dysfunction can result when services are developing from the second to the third stage at the same time that the board is developing from stage 1 to 2. Moreover, the board development can be dysfunctional when it gets ahead of the administrative
177
infrastructure—the board can end up micromanaging unless there is confidence that the administration is capable. Asynchronous development in the five features has provided one target in organizational capacity building efforts—to bring the features into greater alignment. 5.2. Three hypotheses about implementing complex programs with fidelity Developmentally mature agencies are more likely to implement complex programs with fidelity. The observations about agency maturity offer three hypotheses about why this is the case. 1. In more highly developed agencies, staff are more differentiated vertically and horizontally by specialization, enabling those with the greatest expertise in a particular component of the program to concentrate on that component, thus improving the chances that a program model can be implemented. For example, these roles are differentiated in some well-known complex programs: a. intake worker, case manager; counselor; b. dispatcher, paramedic, emergency room physician, recovery team, cardiologist, cardiac rehabilitation staff; c. reading teacher, teacher’s aide, school librarian. 2. In more highly developed agencies, staff are more differentiated vertically by expertise. Therefore, supervisors are more likely to have the skills necessary to effectively oversee implementation by their staff. This may enhance service delivery and therefore, fidelity to a program model. 3. In less highly developed agencies, staff are typically responsible for more than one function in the organization. They are also forced to cross-subsidize many activities, thus dividing staff time and attention. It is important to emphasize that we are not speaking of ‘better’ or ‘worse’ organizations to implement programs. The trade-offs are between speed and responsiveness in the less formally developed organization, versus higher order functioning in the more formally developed organization—that is, specialization within increasingly narrow areas of expertise. We would not say a brain surgeon is ‘better’ than a primary care physician; in fact for many tasks, like initial diagnosis or triage, we would prefer the less specialized practitioner. In the same way, the more developed organization permits greater specialization, but the less developed organization may be more suitable to provide some services. 5.3. Is this framework necessary for learning? Organizational size, age, and annual operating budget are conventional indicators of organizational development. In using the framework, however, we have amassed considerable evidence that static and arbitrary criteria such as these are as insufficient to understand the maturity of non-profit service agencies. We have observed some agencies with operating
178
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
budgets of $10 million that are fairly undeveloped in terms of the features of the framework, while some agencies of $2 million can be more developed. Old established agencies are ‘mature’ in a temporal sense, but not necessarily in their capacity or development. Organizational features develop at different rates and in different patterns, just as humans grow and develop in ways that are not adequately explained by age and size. Those using the SOC scoring rubric have come to recognize that how an organization represents itself to the world may not reflect actual maturity. Agencies are acutely aware that program planners and funders believe they ‘should’ operate formally, and will present themselves this way even if the reality is very different. Many non-profits try to look bigger and more mature than they are in order to attract funding. One exception to this rule was an agency with a very modest operating budget. As assessment proceeded, it emerged that the agency had a large, profitable subsidiary that did not come up in initial conversations about financial resources. When asked why they did not initially mention the subsidiary, the staff said they were worried that people would stop giving to the agency if they knew about it. Checklists and self-assessments of non-profit capacity abound. They may be suitable for superficial monitoring, but have not been tested for validity and reliability and are not suitable for disciplined inquiry into agency development and capacity. We have observed these self-assessments actually distract agency staff from the real issues at hand, because staff will focus on issues that they believe are important to funders. Checklists and self-assessments also assume that agency staff has sufficient expertise about the development of their own organizations that they can always articulate what their real needs are. By contrast, the preferred raters for the SOC rubric have extensive experience with non-profit organizations and development. Checklists and self-assessments are not sufficient when the purpose is to diagnose organization dysfunctions, when the goal is to identify key areas for organizational growth, when changes in capacity require sensitive measurement, or whenever a deeper understanding of organizational maturity is required. 6. Conclusion The SOC framework is useful for understanding the maturity of non-profit agencies, detecting changes in maturity, designing developmentally appropriate capacity building interventions, and comparing agencies on the basis of developmental maturity. One of the more useful features of the framework has been to capture the general direction of development (and devolution) of non-profit agencies, and to use that as a basis to for inquiry about forces underlying change. Acknowledgements This paper was made possible by grants from the Robert Wood Johnson Foundation to the University of Pittsburgh.
The authors would like to thank John Stevenson and Vincent Francisco for their early suggestions about the development of the instrument. References Berwick, D. M., Godfrey, A. B., & Roessner, J. (1991). Curing health care: New strategies for quality improvement. San Francisco, CA: Jossey-Bass. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. Downs, A. (1967). Inside bureaucracy. Boston, MA: Little Brown. Dreyfus, H. L., Dreyfus, S. E., & Athanasiou, T. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York: Free Press. Edelstein, M. R., & Wandersman, A. (1987). Community dynamics in coping with toxic contaminants. In I. Altman, & A. Wandersman (Eds.), Neighborhood and community environments. New York: Plenum Press. Ericsson, K. A., & Smith, J. (Eds.). (1991). Toward a general theory of expertise: Prospects and limits. Cambridge: Cambridge University Press. Frick, T., & Semmel, M. I. (1978). Observer agreement and reliabilities of classroom observational measures. Review of Educational Research, 48, 157–184. Greenlee, J. S., & Trussel, J. M. (2000). Predicting the financial vulnerability of charitable organizations. Nonprofit Management and Leadership, 11, 199–210. Hall, A. D., & Fagen, R. E. (1968). Definition of system. In W. Buckley (Ed.), Modern systems research for the behavioral scientist: A sourcebook. Chicago, IL: Aldine. Hanks, S. H., Watson, C. J., Jansen, E., & Chandler, G. N. (1994). Tightening the life-cycle construct: A taxonomic study of growth stage configurations in high technology organizations. Entrepreneurship Theory and Practice , 5–29. Harrison M. I. (1987). Diagnosing organizations: Methods, models, and processes. Applied social research methods series, Vol. 8. Newbury Park, CA: Sage. Leviton, L., Herrera, C., Pepper, S.K., Fishman, N. W., & Racine, D.P. (2006). Faith in action: Capacity and sustainability of volunteer organizations. Evaluation and Program Planning, 29(2), doi:10.1016/2006.01.011 Hoffman, R. R. (Ed.). (1992). The psychology of expertise: Cognitive research and empirical AI. New York: Springer. Hunter, D. E. K. (2006). Using a theory of change approach to build organizational strength, capacity and sustainability with not-for-profit organizations in the human services sector. Evaluation and Program Planning, 29(2), doi:10.1016/j.evalprogplan.2005.10.003. Hunter, D. E. K., & Koopmans, M. (this issue). Calculating program capacity using the concept of active service slot. Evaluation and Program Planning, 29(2), doi:10.1016/j.evalprogplan.2006.10.002. Katz, D., & Kahn, R. L. (1966). The social psychology of organizations. New York: Wiley. Kazanjian, R. K., & Drazin, R. (1990). A stage-contingent model of design and growth for technology-based new ventures. Journal of Business Venturing, 5, 137–150. Lawrence, P. R., & Lorsch, J. W. (1969). Developing organizations: Diagnosis and action. Reading, MA: Addison-Wesley. Lesgold, A. (2003). Detecting technology’s effects in complex school environments. In G. Haertel, & B. Means (Eds.), Rigorous research designs to evaluate technology effects. New York: Teachers College Press. Letts, C. W., Ryan, W. P., & Grossman, A. (1999). High performance nonprofit organizations: Managing upstream for greater impact. New York: Wiley. Leviton, L. C., Needleman, C. E., & Shapiro, M. A. (1998). Confronting public health risks: A decision maker’s guide. Thousand Oaks, CA: Sage.
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179 Light, P. C. (1998). Sustaining innovation: Creating nonprofit and government organizations that innovate naturally. San Francisco, CA: Jossey-Bass. Miller, R. L., Bedney, B. J., & Guenther-Grey, C. (2003). Assessing community capacity to provide HIV prevention services: The feasibility, evaluability, sustainability assessment protocol. Health promotion practice. Health Education and Behavior, 30(5), 582–600. Quinn, R. E., & Cameron, K. (1983). Organizational life cycles and shifting criteria of effectiveness: Some preliminary evidence. Management Science, 29(1), 33–51.
179
Schuh, R. G. (2005). Evaluation of the small agency building initiative: Final report. Pittsburgh, PA: University of Pittsburgh. Simon, H. A. (1974). How big is a chunk? Science, 183, 482–488. Simonton, D. K. (1996). Creative expertise: A life-span developmental perspective. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports and games (pp. 227–253). Mahwah, NJ: Lawrence Erlbaum. Tuckman, H. P., & Chang, C. F. (1991). A methodology for measuring the financial vulnerability of charitable nonprofit organizations. Nonprofit and Voluntary Sector Quarterly, 20(4), 445–460.
A framework to assess the development and capacity of non-profit agencies Russell G. Schuh a,*, Laura C. Leviton b a
Department of Administrative and Policy Studies, School of Education, University of Pittsburgh, 5906 Wesley W. Posvar Hall, Pittsburgh, PA 15260, USA b Research and Evaluation, Robert Wood Johnson Foundation, Route 1 North and College Road East, Princeton, NJ 08543, USA Received 1 November 2005; received in revised form 15 December 2005; accepted 30 December 2005
Abstract In this article, we present a framework to assess non-profit agencies’ levels of development and capacity. The framework is useful for evaluation because it helps to predict and explain program implementation. The framework is useful for program planning because planners must select organizations that are suitable to implement programs, and sometimes they must build organizations’ capacity to do the work. A scoring rubric called the stages of organizational change measures development and capacity, using the maturity modeling approach. Maturity modeling is a technique that emerged from the worlds of business and technology. ‘Maturity’ does not imply value judgments about organizations; rather, the level of development and capacity should fit the services to be delivered. The scoring rubric has good reliability and validity and has been applied in three ways: (1) to assess agency capacity at a single time point; (2) to describe capacity development; and (3) to evaluate capacity building initiatives. q 2006 Elsevier Ltd. All rights reserved.
1. Introduction Among all the forces affecting program implementation and survival, the organizational context has received surprisingly little formal consideration. ‘Formal consideration’ means the exploration and measurement of generic organizational features that predictably affect programs. Programs are not the same things as the agencies that house them, yet evaluation has failed to formally consider both the organization’s stage of development and the organizational characteristics that support the capacity necessary for non-profit agencies to do their work well over time. Formative or implementation evaluations inherently must address questions of organizational capacity. Programs are not some idealized models and they do not take place in Petri dishes. They are implemented largely within organizations, which have strengths and weaknesses, other activities, and other agendas. The literature on organizations and systems reveals that some of these forces are commonly understood, and others are becoming better understood. Insights about the characteristics of non-profit agencies (such as leadership and governance, organizational depth, and financial sustainability) can help us predict the quality of * Corresponding author. Tel.: C1 412 624 7659; fax: C1 412 648 1784. E-mail addresses: [email protected] (R.G. Schuh), [email protected] (L.C. Leviton).
0149-7189/$ - see front matter q 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2005.12.001
program implementation, explain outcomes, and inform the external validity of outcome conclusions. In this article, we present a framework for considering how non-profit agencies’ development and capacity affect program implementation.1 The framework has been applied to 56 nonprofit agencies to date. It has been used in three ways: (1) to assess agency capacity at a single time point; (2) to describe capacity development; and (3) to evaluate capacity building initiatives. First, we present an initial orientation to the framework and definitions of terms, beginning to relate the framework to evaluation and program planning. We then review the empirical and theoretical underpinnings of the framework, along with the method we have developed for rating organizational features according to the framework. Finally, we provide some initial insights from the framework that may assist evaluation and program planning. 2. Basic orientation and definitions Non-profit agencies differ greatly in the degree to which they have developed five key organizational features: Governance, financial resources, organizational development (i.e. structure and differentiation), internal operations, and core services. These five features work through two levels of 1
Governmental agencies are also at different stages of development in ways similar to the ones studied here. However, the framework has not been formally applied to such agencies as of this writing.
172
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
organizational capacity—individual expertise and organizational procedures and resources (see definitions below). Although programs are heterogeneous in terms of services and populations served, by contrast development of these five organizational features at two levels of capacity appear to be generic to all non-profit agencies that provide services. The constructs and theories underlying the framework are the focus of this article. They are based on precepts from diverse sources of which the most important are maturity modeling, organizational life-cycle theory, systems theory, cognitive psychology, and educational research. These five organizational features are being explored using a scoring rubric, the Stages of Organization Capacity (SOC), designed to assess agencies’ stages of development. The SOC relates theoretical constructs to observable attributes within each feature, by rating their stages of development and capacity. The framework, rather than the SOC rating instrument, is the focus of this article. Instrumentation is satisfactory at this stage, but the SOC continues to undergo revision for a very important data-based reason. The observations derived from the SOC are starting to teach us new things about the nomological net: that is, the set of underlying relationships among the features and changes in the features of the SOC. This is the sign of a useful instrument, and is predicted as one of the criteria for construct validation (Cronbach & Meehl, 1955). The nomological net provides a framework for relating observable attributes to each other. Both the framework and SOC are demonstrating their value in helping us improve our understanding of the fate of programs and services within non-profit agencies. To our knowledge, most of these dynamics have not been described in the research literature on non-profit agencies or evaluation—a significant gap in the literature about non-profit organizations and program evaluation that we are making an initial effort to fill in with this paper. 2.1. Definition: non-profit organizational capacity We have settled on the following operational definition of organizational capacity: the ability to successfully implement and complete a new project or to expand an existing one successfully. Other operational definitions are reasonable, but this one focuses on characteristics that are amenable to change and that affect non-profits’ ability to carry out services. An organization’s ‘projects’ are not limited to service delivery, but can include a fund raising drive, public relations program, and other non-program, non-service delivery efforts. When agencies are out of equilibrium (i.e. expanding or spiraling down), need quick response to volatile environments, take on new work, or expand their missions, organizational capacity and capacity building become issues. An organization is analogous to a glass of water, and its capacity is analogous— not to the size of the glass—but to how much more water the glass can carry without spilling some. The research literature to date suggests two levels of organizational capacity that, when enhanced or more fully
developed, tend to improve the likelihood of successful program implementation (Letts, Ryan, & Grossman, 1999): † Individual expertise that includes the skills, knowledge, and experience that employees and volunteers bring to the organization (in this framework ‘leadership’, is a type of expertise, to be expressed in a particular organizational context). † Organizational resources and procedures that permit agencies to use individual expertise productively. Greater organizational differentiation will assist individuals to work at a higher level, using more specialized skills. An example is the use of support staff to handle repetitive work, freeing more skilled staff to tackle the more complex tasks. Less developed agencies are unable to function at the same level of specialization. When capacity building is the goal of interventions, too often the focus is on individual training, to enhance expertise and make higher order task performance possible. Yet, the organization may not be able to use the increased expertise without some changes in its own processes and resources. This pattern is abundantly clear in the literature on medical care quality improvement, where the focus has shifted to improving the processes of medical care because individual intervention has small effects (Berwick, Godfrey, & Roessner, 1991). 2.2. Definition: non-profit organization development We define development in terms of the level, or stage, of five key features that comprise the framework. They were selected because, according to the literature, they: (a) are common to all non-profit agencies, (b) are likely to change as agencies develop, (c) can develop at different rates, and (d) affect an organization’s ability to implement new or expanded community innovations. We chose these features based on evidence from the organizational life-cycle literature suggesting that agencies have a general pattern of maturation in common, and that these features vary with organizational stage of development. Quinn and Cameron (1983) present evidence that the same patterns emerge in a wide range of settings, structures, and functions. The five features are: 1. Governance—legal status of organization and associated documents; board composition, functioning, and development; relationship between board and staff. 2. Financial resources—organization’s funding history and its ability to generate funds and to sustain itself; funding resiliency, diversity, and vulnerability. 3. Organizational development—degree of horizontal and vertical structure of the organization; administrative versus service role differentiation; skills and experience of administrative staff; role differentiation and specialization 4. Internal operations—degree to which an organization has differentiated its administrative functions, degree to which tasks are specialized by degree of skill or training they require. Often internal operations are considered overhead
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
or support operations (e.g. executive leadership, financial systems, human resource systems, and management information systems (MIS)). 5. Core services—degree to which services (or key businesses) of an organization are differentiated and the skill and experience of staff in providing them.
2.3. Relevance to evaluation and program planning 2.3.1. Program capacity versus organizational capacity The program to be evaluated is different from the agency that houses it. Thus, program capacity needs to be distinguished from organizational capacity (see Hunter, 2006; Hunter & Koopmans, 2006). The latter relates to agencies’ ability to expand or to start something new. As described below, the feature of core services relates most closely to program capacity, except that many non-profit agencies have more than one core service. Apparent problems of program capacity, such as slow start up of a program or faulty implementation of service delivery, are sometimes symptoms of organizational capacity problems. Planners and funders often believe that capacity is impaired when agencies lose a program or a funding stream. However, in using the framework we have observed quite the opposite: challenges to organizational capacity are greatest when a new program begins, not when it is ended. At the beginning, staff must be hired or diverted from other functions; new organizational procedures must be put in place, and sometimes a new core service (or core business) must be introduced. In a real sense, organizational capacity is greatest when a program is at its end. Staff is available for re-assignment, procedures have been worked out, and the program has been integrated (more or less) into the overall conduct of the non-profit agency’s work. 2.3.2. The relationship between non-profits’ development and capacity Capacity and development do not necessarily go hand in hand. Highly developed agencies that are already operating at full capacity can experience capacity problems when they add something new, just as less developed agencies can. The difference is that, if less developed agencies are to generate additional capacity, they will often need to become more formally developed. Programs are often planned and funded that require an agency to use more formal procedures and methods than their stages of development can support. For example, with a new or expanded program, the payroll might increase from five staff to 25, creating a need for more formally developed internal operations. The funder of the new program might require a more formal financial system and more formal performance reporting than the agency has in place. More formal systems require greater effort on the part of staff to complete records and follow procedures, and less developed agencies often need to grow these organizational features at the same time that they are expanding services or taking on a new
173
program, which can produce predictable stresses both on these features and on the programs. 2.3.3. Challenges to value judgments At this point in the exposition, we need to confront some implicit value judgments about organizational development and capacity. Development does not connote ‘better’ or ‘worse’ services. Rather, there are trade-offs for providing services. Both informally and more formally developed agencies have their places in service delivery. Agencies implement services differently depending, in part, on their degree of formal development. Agencies that operate more informally can have advantages for certain services: for other purposes a more developed organization will be appropriate because it can better accommodate a particular program. Some agencies should remain informal precisely because they fill an important niche and do not require the kind of development that is often a focus of funders (see for example, Leviton, Herrera, Pepper, Fishman, & Racine, 2006). In fact, there are sound reasons for some agencies to reject more formal development. Small, horizontal agencies can be more innovative than larger, more formal agencies—they have more capacity for rapid innovation. Grassroots agencies are often more sensitively responsive to their environments, as seen in the many examples of communities’ need to cope with exposures to toxic environmental agents or to serve people with AIDS. Bureaucracies could not adapt in a timely way to assist or to meet the community on its own terms—they had limited capacity to evolve and address their needs (Edelstein & Wandersman, 1987; Leviton, Needleman, & Shapiro, 1998). Some grassroots agencies can provide more effective services to community members, compared to more developed agencies. They have more capacity to provide these services because their staffs possess expertise about the community, and the organization’s procedures permit its expression. More formally developed agencies will tend to have fewer problems implementing programs under at least three conditions: when the program is new, when it is expanding (as described above); or when a pre-specified, complex program model must be implemented with fidelity. The problem of implementing with fidelity has been encountered in many programs. Recently, program planners and funders have started to look for agencies that are sufficiently developed to implement the models with fidelity (Miller, Bedney, & Guenther-Gray, 2003). The framework offers three testable hypotheses about such agencies, as seen at the end of this article. 3. Development of the framework 3.1. Maturity models to assess agencies Agencies are fluid and dynamic. From our observations, they can sometimes develop or devolve quite rapidly. Based on the literature concerning organization life-cycles, it seems important to measure stages of development. The research literature points to a useful approach to capture these
174
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
dynamics: the maturity model. A maturity model involves a set of features and a related set of levels or stages for each feature (Lesgold, 2003). The stages or levels represent a developmental continuum arranged from initial, or beginning, condition to mature, or fully developed, condition. Maturity models extend evaluators’ abilities to track change in three important ways: (a) they can be developed for any number of features, (b) they emphasize the development of scoring rubrics to facilitate reliable assignment of each feature to the most appropriate stage or level, and (c) they have already been used extensively to assess organizational-level innovations in educational and technology transfer environments (Kazanjian & Drazin, 1990). ‘Maturity model’ is a technical term. It does not imply a value judgment or any prediction about the direction that capacity or development will take. The flexibility of maturity models for assessing both individual- and organization-level attributes makes them ideal for addressing the complex issues of agency capacity. A scoring rubric is a taxonomy of observed characteristics or symptoms. The taxonomy can accrue, becoming more differentiated through development and use, and therefore more reliable and precise (Frick & Semmel, 1978). The stages or levels of the features are arranged from those one would expect to find in a less mature organization to those one would expect in the most mature organizations. Each feature has attributes for which scoring rubrics have been developed. In developing the SOC scoring rubric, an iterative process was used, consisting of (a) identification of the core set of organizational attributes from the literature that were postulated to change as an organization matured, followed by (b) retrospective application to an organization with data on file from previous work, followed by (c) revision as indicated by the retrospective application, and (d) application of the revised instrument to another case. The maturity model continued to be refined by the addition of attributes as experience was gained in the field (both calibrated and uncalibrated at the present time). To illustrate the scoring rubric, in Fig. 1 we present some examples of attributes for the feature of financial resources (the soc scoring manual is available from the first author). As seen in these examples, the staging of attributes is done on the basis of specific anchor descriptions, wherever possible. In other words, the description typifies the attribute at a specific stage of development. Uncalibrated indicators will often point to movement from one stage to another. In other words, these would provide evidence that the organization is developing more mature features, or devolving to less mature features. Such movement provides a basis for further inquiry about the agency. 3.2. Assessing individual expertise The SOC rubric uses two levels of maturity modeling: one for organizational processes and one for individual expertise. A research literature on individual expertise, derived from cognitive psychology, applies to the non-profit setting in a straightforward way. Expertise develops in stages (Ericsson & Smith, 1991; Hoffman, 1992; Simon, 1974). ‘Expertise’ is
often described only as possessing extensive training and experience in a given field. However, it is possible for individuals to possess both extensive experience and training in a field and, at the same time, fail to achieve expert performance. Expertise goes beyond experience and training, to learning to understand and solve problems in different ways, a skill that grows to be intuitive in a true expert. We adapted a five-stage model from the work of Dreyfus, Dreyfus, and Athanasiou (1986): Stage 1 Novice. Uses simple rule-based problem solving; uses ‘backward thinking’ to envision a solution and follows rules to get there. Stage 2 Advanced beginner. Uses backward thinking with a greater number of rules; situational thinking begins to emerge. Stage 3 Competent. Uses situational problem solving (different rules for different circumstances); can adopt a perspective and follow it to a solution. Stage 4 Proficient. Uses multiple perspectives, rapidly assessing which to adopt; understands relevant data; forward thinking emerges. Stage 5 Expert. Uses intuitive forward-thinking problem solving; recognizes underlying patterns. Expertise is domain-specific and requires years of practice to develop (Simonton, 1996). For example, in using the framework we have encountered professionals who were very proficient service providers, but then became inexperienced managers. As managers they started over as novices, or at best merely competent, because expertise is domain-specific. The SOC has also revealed how organizational limitations of capacity effectively reduce the expertise of highly proficient people. For example, an information technology professional was trying to develop an interagency information system. However, he was reduced to entering data and fixing network problems, lower order tasks for which he had no staff support. 3.3. Data collection and validation One or more outside observers use the scoring rubric to produce a profile of organizational stage of development. Ideally, observers have extensive experience in studying or working with agencies and their behavior, because such individuals will more readily observe underlying patterns. Developing the profile of a non-profit organization occurs in stages. Data collection for SOC is unstructured, qualitative, and iterative—that is, it goes in cycles. Quantification and structure come later, after each round of data collection, once the rater’s impressions are applied to the scoring rubric. This then becomes the basis of further inquiry to fill out a more detailed profile. Not every attribute of every scoring rubric has to be rated to gain an overall picture of development for each of the five features. The level of detail is dictated by the purpose of data collection. Reliability and validity information is accruing along with the taxonomy. To date, various versions of the scoring rubrics
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
175
Agency Stage of Development
Features Stage 1 Initial, start-up or small agency Financial Resources Maturity (sample attributes)
C
Stage 3 Established agency
Stage 4 Proficient agency
Stage 5 Highly productive agency
Unstable, unpredictable or insecure funding
Limited funding, more stable than stage 1 agency
Established agency funding where organization has some control over its own destiny
Stable sources of income or funding
Highly stable funding
Opportunistic funding (like that resulting from a new or emerging policy concern).
Contingency funding for activities or services but not long term (e.g. summer youth activities).
Initiative funding Project funding derived from services related to an on-going policy priority like violence prevention or early childhood development.
Niche funding Agency funded through on-going source like United Way or local funding authority (e.g. mental health, housing or juvenile justice).
Institutionalized funding for at least a core of services or programs (e.g. Funding is a line item on a state or local authority budget.
Start-up or seed money only or short term funding.
Projects or activities funded on a one-time basis (funding may span several years).
Funding may be for specified periods but contains provision for renewal or continuation.
Core funding ongoing, may have provision for periodic review or renewal.
Core service funding is highly stable. May be supported by spinoff services or products.
A
B
Stage 2 Advanced young or small agency
Fig. 1. Examples of scoring rubric for financial resources.
have been applied to 56 agencies. Interrater reliability (kappa statistic, not percent agreement) ranges from 0.81 to 0.86 for the features of governance, financial resources and organizational development.2 Interrater reliability (kappa) for the features of internal operations and core services are currently in the 0.6 ranges. The difference may be due, in part, to availability of literature pointing to common principles for the first three features, which allowed comparison across organizations. No such literature was available for internal operations or core services, so the attributes presented are all original work and developed ‘from scratch’. Because many relevant features and attributes are derived from the literature on non-profit agencies and organizational life-cycles, we have some confidence in the construct validity of the framework and SOC scoring rubric. Furthermore, the emerging relationships as assessed by the SOC scoring rubric provide some confidence in the utility of the framework for understanding the nomological net. Finally, face validity of the SOC scoring rubric is quite high, because executive directors and board members have all indicated that the SOC profiles reveal good understanding of the agencies’ stage of maturity. 4. The framework in detail 4.1. Governance Governance strongly affects other features of interest to program planning and evaluation. Governance includes the 2 Kappa statistics adjust for the probability of a given response and are therefore preferable to percent agreement.
board of directors and the organization’s legal relationship with the environment, including charter, type of organization, and mission. Boards of new and less developed agencies tend to be small and operate informally. As agencies mature, boards tend to grow in size and become more formal in their structure and work. Board focus also changes as agencies develop, from rules-based requirements of getting established, to stabilization of the organization’s resources, and finally to growth and expansion of mission. The relationship between boards and organization staff also changes as agencies develop. Staff leadership in early stages evolves into greater differentiation between the functions of management (by staff) and policy (by board). Development of this domain was based on literature from systems theory, particularly open systems theory, and organizational development (Hall & Fagen, 1968; Katz & Kahn, 1966). Board development can be a focus in organizational capacity building, as in a recent initiative of the Robert Wood Johnson Foundation (RWJF) evaluated by the first author (Schuh, 2005). The evaluation revealed very little change in boards due to this initiative. The reasons may be explained, in part, by certain attributes of the agencies. In general, as agencies mature, their governing boards focus increasingly on organization development and growth. However: 1. Several agencies had mandates to include consumers, public officials, or representation by other agencies. Boards with mandated representation may retain a greater interest in advocating for their constituency (a legitimate focus), than in growing the organization.
176
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
This feature addresses the organization’s ability to secure resources for sustainability and mission development. Agencies can progress from unstable to highly stable funding, although they may lose ground from shifts in the environment. Research on the non-profit sector, management, and organization development has contributed to this domain. For example, it is clear that sustainability is related, in part, to organizational performance (Letts et al., 1999; Light, 1998). The relationship to implementation should be readily apparent: if an agency cannot meet the payroll, it is unlikely the program can be well-implemented or sustained in that agency. Financial resources also affect the degree to which agencies must crosssubsidize other organizational functions using program resources. Financial maturity is an important subset of attributes under this feature. It is reflected in routinized or formalized activities to secure funds: for example, provisions for recovering agency overhead costs, yearly fund raising campaigns, or new product development. More financially mature agencies can afford to choose funding sources that will pay for activities that are in line with their mission, rather than on an opportunistic basis. Fund raising that reflects greater financial maturity is seen in an absence of crisis orientation (e.g. to meet the payroll or the rent), responsibilities shared by the staff and the board, routinized or predictable fund raising activities, and a more diverse or strategic set of fund raising activities. A second important subset concerns financial vulnerability, the ability to stay in operation despite a variety of financial stresses or agency transitions. Vulnerability is reflected in whether an agency has equity, a diversity of funding sources and types of funders (Greenlee & Trussel, 2000; Tuckman & Chang, 1991), adequate operating margins, and defined administrative costs that support the organization’s infrastructure. These are all important predictors of organization survival (see also Herrera et al., this issue).
(often called vertical and horizontal differentiation) change with development and growth. Tall organizational structures are often older and more formal than short ones, and communicate more formally as well (Downs, 1967; Quinn & Cameron, 1983). Short structures can be an indication of less developed administrative infrastructure (Hanks, Watson, Jansen, & Chandler, 1994). In general, non-profit agencies tend to under-emphasize infrastructure development—some would say they starve it because of the priority placed on providing services (Letts et al., 1999). In less developed agencies, the roles of administration and of service delivery are often undifferentiated, with agency leaders ‘wearing several hats’. Administration and service roles become differentiated as agencies develop infrastructure and capacity. Larger and more differentiated administrative infrastructures are often an indicator of highly productive and financially robust agencies. Such agencies, in turn, tend to have greater administrative specialization, because the infrastructure helps people do the work that requires their highest skill levels most of the time. For example, a more vertically differentiated organization will have specialists in human resources, budget, communications, fund raising and management information systems. By contrast, in less developed agencies, it is common to find the executive director cleaning offices, sending notices to the board, and at the same time preparing the financial report. It is also common to find executive directors providing direct services, conducting fund raisers, and serving as the human resources officer. Since expertise is highly domain specific, working in a variety of roles precludes application of one’s greatest expertise much of the time. Observations about role specialization and administrative differentiation provide insights about the implementation of programs. For example, greater administrative differentiation is reflected in greater overhead costs, so a comparable program will appear to cost more in more developed agencies than in less developed organizations. However, the savings in less developed agencies may not be real, because many of the overheadsupported functions found in more mature agencies are supported through cross-subsidization in less mature agencies. Undifferentiated agencies tend to characterize themselves as having cultures in which ‘everyone does everything’ or they may say ‘we wear many hats’. Usually only one hat is supported from the outside, or at least some hats will not be supported. This is a form of cross-subsidization. The extent of cross-subsidization limits an agency’s capacity to take on additional projects, or to implement a program with fidelity—both of which can be very significant opportunity costs.
4.3. Organization development
4.4. Internal operations
This feature addresses the manner in which agencies structure themselves to achieve their missions. It has been informed by organizational, management, and non-profit literature (Harrison, 1987; Lawrence & Lorsch, 1969; Letts et al., 1999). The height and width of the organization chart
Internal operations in non-profit agencies are devised over time, and there can be great variation in these arrangements. However, each internal operation can be compared to others in terms of the developmental maturity of their infrastructure (space, administrative components, and logistic support);
2. Some agencies were sub-units of parent organizations, and under these conditions the agency board has limits on its ability to influence direction and policy. 3. Most of the agencies suffered from ‘founder’s syndrome’ in which an organization cannot develop beyond the vision of a strong leader–founder. Since boards tend to focus more on operations as agencies mature, the presence of a strong founder hindered their transition to a focus on operations. We find references to founder’s syndrome in the literature, but no explanation of the underlying dynamics.
4.2. Financial resources
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
technology (computers, specialized equipment); role differentiation (horizontal separation of tasks by domain); role specialization (vertical separation of tasks by skill within a domain); and expertise of those providing the internal operation. This is also the approach employed to score the maturity of core services, because it accommodates great variation in organizational arrangements while permitting comparison across agencies and within agencies across time. The internal operations that are the framework’s current focus include management information systems (MIS), which can be informal, as with notes on the back of a napkin, or formal as with computerized MIS. In the same way, financial operations can be informal, as in writing checks by hand, balancing a checkbook, and issuing ad hoc reports. It can be formal, as with computerized monitoring of project expenditures and regularly generated balance sheets and income projections. The attributes of human resources, communications, and development (as in product development and fundraising) all reflect agency stage of development. The rubric can be expanded to issues such as planning, where necessary. 4.5. Core services Within core services, SOC raters score each separate project. Projects can reflect different organizational activities, but we also distinguish more than one project when different funding sources require different things from the same activity. This gets to the specifics of what the agency is implementing at the street level. For example, drug treatment may be funded through block grants from the state, or directly by Medicaid, and the requirements shape the specific interventions. Core services largely define a community agency, which is often founded with the idea of addressing a perceived need. Services, like internal operations, are devised over time and can vary infinitely. As with internal operations, certain attributes appear consistently, offering comparison across agencies and within a developing agency across time. The attributes of core services, like those of internal operations, include infrastructure, technology, role differentiation and specialization, and expertise of the key service providers. 5. Learning from the framework 5.1. Patterns of growth and diagnosis of capacity needs Experience to date leads us to observe a general pattern: core services tend to develop first, followed by some aspects of financial management. What is surprising is that development of administrative infrastructure often lags by a stage and that board development and functioning generally lags even more. From observation, it appears that significant dysfunction can result when services are developing from the second to the third stage at the same time that the board is developing from stage 1 to 2. Moreover, the board development can be dysfunctional when it gets ahead of the administrative
177
infrastructure—the board can end up micromanaging unless there is confidence that the administration is capable. Asynchronous development in the five features has provided one target in organizational capacity building efforts—to bring the features into greater alignment. 5.2. Three hypotheses about implementing complex programs with fidelity Developmentally mature agencies are more likely to implement complex programs with fidelity. The observations about agency maturity offer three hypotheses about why this is the case. 1. In more highly developed agencies, staff are more differentiated vertically and horizontally by specialization, enabling those with the greatest expertise in a particular component of the program to concentrate on that component, thus improving the chances that a program model can be implemented. For example, these roles are differentiated in some well-known complex programs: a. intake worker, case manager; counselor; b. dispatcher, paramedic, emergency room physician, recovery team, cardiologist, cardiac rehabilitation staff; c. reading teacher, teacher’s aide, school librarian. 2. In more highly developed agencies, staff are more differentiated vertically by expertise. Therefore, supervisors are more likely to have the skills necessary to effectively oversee implementation by their staff. This may enhance service delivery and therefore, fidelity to a program model. 3. In less highly developed agencies, staff are typically responsible for more than one function in the organization. They are also forced to cross-subsidize many activities, thus dividing staff time and attention. It is important to emphasize that we are not speaking of ‘better’ or ‘worse’ organizations to implement programs. The trade-offs are between speed and responsiveness in the less formally developed organization, versus higher order functioning in the more formally developed organization—that is, specialization within increasingly narrow areas of expertise. We would not say a brain surgeon is ‘better’ than a primary care physician; in fact for many tasks, like initial diagnosis or triage, we would prefer the less specialized practitioner. In the same way, the more developed organization permits greater specialization, but the less developed organization may be more suitable to provide some services. 5.3. Is this framework necessary for learning? Organizational size, age, and annual operating budget are conventional indicators of organizational development. In using the framework, however, we have amassed considerable evidence that static and arbitrary criteria such as these are as insufficient to understand the maturity of non-profit service agencies. We have observed some agencies with operating
178
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179
budgets of $10 million that are fairly undeveloped in terms of the features of the framework, while some agencies of $2 million can be more developed. Old established agencies are ‘mature’ in a temporal sense, but not necessarily in their capacity or development. Organizational features develop at different rates and in different patterns, just as humans grow and develop in ways that are not adequately explained by age and size. Those using the SOC scoring rubric have come to recognize that how an organization represents itself to the world may not reflect actual maturity. Agencies are acutely aware that program planners and funders believe they ‘should’ operate formally, and will present themselves this way even if the reality is very different. Many non-profits try to look bigger and more mature than they are in order to attract funding. One exception to this rule was an agency with a very modest operating budget. As assessment proceeded, it emerged that the agency had a large, profitable subsidiary that did not come up in initial conversations about financial resources. When asked why they did not initially mention the subsidiary, the staff said they were worried that people would stop giving to the agency if they knew about it. Checklists and self-assessments of non-profit capacity abound. They may be suitable for superficial monitoring, but have not been tested for validity and reliability and are not suitable for disciplined inquiry into agency development and capacity. We have observed these self-assessments actually distract agency staff from the real issues at hand, because staff will focus on issues that they believe are important to funders. Checklists and self-assessments also assume that agency staff has sufficient expertise about the development of their own organizations that they can always articulate what their real needs are. By contrast, the preferred raters for the SOC rubric have extensive experience with non-profit organizations and development. Checklists and self-assessments are not sufficient when the purpose is to diagnose organization dysfunctions, when the goal is to identify key areas for organizational growth, when changes in capacity require sensitive measurement, or whenever a deeper understanding of organizational maturity is required. 6. Conclusion The SOC framework is useful for understanding the maturity of non-profit agencies, detecting changes in maturity, designing developmentally appropriate capacity building interventions, and comparing agencies on the basis of developmental maturity. One of the more useful features of the framework has been to capture the general direction of development (and devolution) of non-profit agencies, and to use that as a basis to for inquiry about forces underlying change. Acknowledgements This paper was made possible by grants from the Robert Wood Johnson Foundation to the University of Pittsburgh.
The authors would like to thank John Stevenson and Vincent Francisco for their early suggestions about the development of the instrument. References Berwick, D. M., Godfrey, A. B., & Roessner, J. (1991). Curing health care: New strategies for quality improvement. San Francisco, CA: Jossey-Bass. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. Downs, A. (1967). Inside bureaucracy. Boston, MA: Little Brown. Dreyfus, H. L., Dreyfus, S. E., & Athanasiou, T. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York: Free Press. Edelstein, M. R., & Wandersman, A. (1987). Community dynamics in coping with toxic contaminants. In I. Altman, & A. Wandersman (Eds.), Neighborhood and community environments. New York: Plenum Press. Ericsson, K. A., & Smith, J. (Eds.). (1991). Toward a general theory of expertise: Prospects and limits. Cambridge: Cambridge University Press. Frick, T., & Semmel, M. I. (1978). Observer agreement and reliabilities of classroom observational measures. Review of Educational Research, 48, 157–184. Greenlee, J. S., & Trussel, J. M. (2000). Predicting the financial vulnerability of charitable organizations. Nonprofit Management and Leadership, 11, 199–210. Hall, A. D., & Fagen, R. E. (1968). Definition of system. In W. Buckley (Ed.), Modern systems research for the behavioral scientist: A sourcebook. Chicago, IL: Aldine. Hanks, S. H., Watson, C. J., Jansen, E., & Chandler, G. N. (1994). Tightening the life-cycle construct: A taxonomic study of growth stage configurations in high technology organizations. Entrepreneurship Theory and Practice , 5–29. Harrison M. I. (1987). Diagnosing organizations: Methods, models, and processes. Applied social research methods series, Vol. 8. Newbury Park, CA: Sage. Leviton, L., Herrera, C., Pepper, S.K., Fishman, N. W., & Racine, D.P. (2006). Faith in action: Capacity and sustainability of volunteer organizations. Evaluation and Program Planning, 29(2), doi:10.1016/2006.01.011 Hoffman, R. R. (Ed.). (1992). The psychology of expertise: Cognitive research and empirical AI. New York: Springer. Hunter, D. E. K. (2006). Using a theory of change approach to build organizational strength, capacity and sustainability with not-for-profit organizations in the human services sector. Evaluation and Program Planning, 29(2), doi:10.1016/j.evalprogplan.2005.10.003. Hunter, D. E. K., & Koopmans, M. (this issue). Calculating program capacity using the concept of active service slot. Evaluation and Program Planning, 29(2), doi:10.1016/j.evalprogplan.2006.10.002. Katz, D., & Kahn, R. L. (1966). The social psychology of organizations. New York: Wiley. Kazanjian, R. K., & Drazin, R. (1990). A stage-contingent model of design and growth for technology-based new ventures. Journal of Business Venturing, 5, 137–150. Lawrence, P. R., & Lorsch, J. W. (1969). Developing organizations: Diagnosis and action. Reading, MA: Addison-Wesley. Lesgold, A. (2003). Detecting technology’s effects in complex school environments. In G. Haertel, & B. Means (Eds.), Rigorous research designs to evaluate technology effects. New York: Teachers College Press. Letts, C. W., Ryan, W. P., & Grossman, A. (1999). High performance nonprofit organizations: Managing upstream for greater impact. New York: Wiley. Leviton, L. C., Needleman, C. E., & Shapiro, M. A. (1998). Confronting public health risks: A decision maker’s guide. Thousand Oaks, CA: Sage.
R.G. Schuh, L.C. Leviton / Evaluation and Program Planning 29 (2006) 171–179 Light, P. C. (1998). Sustaining innovation: Creating nonprofit and government organizations that innovate naturally. San Francisco, CA: Jossey-Bass. Miller, R. L., Bedney, B. J., & Guenther-Grey, C. (2003). Assessing community capacity to provide HIV prevention services: The feasibility, evaluability, sustainability assessment protocol. Health promotion practice. Health Education and Behavior, 30(5), 582–600. Quinn, R. E., & Cameron, K. (1983). Organizational life cycles and shifting criteria of effectiveness: Some preliminary evidence. Management Science, 29(1), 33–51.
179
Schuh, R. G. (2005). Evaluation of the small agency building initiative: Final report. Pittsburgh, PA: University of Pittsburgh. Simon, H. A. (1974). How big is a chunk? Science, 183, 482–488. Simonton, D. K. (1996). Creative expertise: A life-span developmental perspective. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports and games (pp. 227–253). Mahwah, NJ: Lawrence Erlbaum. Tuckman, H. P., & Chang, C. F. (1991). A methodology for measuring the financial vulnerability of charitable nonprofit organizations. Nonprofit and Voluntary Sector Quarterly, 20(4), 445–460.