Constructing a fuzzy-knowledge-based-system: an application for assessing the financial condition of public schools

Expert Systems with Applications 27 (2004) 349–364 www.elsevier.com/locate/eswa

Constructing a fuzzy-knowledge-based-system: an application for assessing the financial condition of public schools Salwa Ammara, William Duncombeb, Bernard Jumpb, Ronald Wrighta,* a

Department of Business Administration, Le Moyne College, Syracuse, NY 13214-1399, USA b Center for Policy Research, Syracuse University Syracuse, NY 13214-1399, USA

Abstract Financial evaluation in the public sector can utilize some of the tools developed for evaluation in the private sector. However, the emphasis on the bottom line characterized by productivity measures does not adequately address all of the issues faced by public institutions. Recent collaborations by researchers in management science and public administration have led to the successful development of an analytical approach that combines fuzzy set theory and knowledge based systems to produce a tool for evaluating the general performance of public institutions. Successful implementations have included evaluations of the management of state governments, the financial administration of large cities, and the credit worthiness of public institutions. This paper describes a recent collaborative project, funded by the State of New York, to develop a system to evaluate the financial condition of the state’s nearly 700 school districts. q 2004 Elsevier Ltd. All rights reserved. Keywords: Fuzzy logic; Knowledge-based-systems; Financial evaluation; Performance evaluation

1. Introduction Performance evaluation and assessment is an integral part of operations both in the private and public sector. Performance evaluation in the private sector has received much attention in the literature (Rossi & Freeman, 1993). Most deal with measures of effective and efficient performances as they relate to the bottom line of institutions (Fried, Lovell, & Schmidt, 1993). These techniques utilize tools traditionally applied to other aspects of operations management. Similar techniques, however, are not as easily available for public institutions. Many agree that performance management is crucial in public institutions and that accountability in public service leads to more effective and efficient performances. However, the use of simple performance measures in evaluating public institutions tends to provide incomplete and often inaccurate evaluations (Ganley & Cubbin, 1992). Performance evaluations of public institutions offer many unique challenges. A useful evaluation process needs to address all these challenges. This paper describes techniques for performance evaluation that directly address some of * Corresponding author. Tel.: þ 1-315-445-4370; fax: þ1-315-445-4787. E-mail address: [email protected] (R. Wright). 0957-4174/$ - see front matter q 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2004.05.004

the difficulties in assessments of public institutions. It also contains several real implementations of these techniques. All of the described applications represent interdisciplinary work combining research by faculty in operations management and in public administration as well as a variety of stakeholders. Much like in the private sector, the primary objective of the evaluation of a public institution is focused on improving performance. In addition, it is important to have the ability to compare institutions that perform similar public services (for example school districts). These comparisons provide benchmarks for performances and allow evaluations to become relevant. Comparisons among public institutions, however, are unique in that they are multifaceted with many qualitative aspects. Such comparisons are not trivial and present the first challenge in the evaluation process. A useful approach must incorporate a mechanism by which comparisons among similar entities can be performed. The objectives of evaluating public agencies also include the ability to provide a framework for describing performance criteria and expectations. More often than not such expectations are subjective in nature and not easily quantifiable. The second challenge in evaluating public agencies is therefore the need for the process to be well

350

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

defined and transparent to the evaluated entity. A useful approach must incorporate a mechanism through which the evaluation process can be clearly described. Performances based on multiple criteria and judgments are particularly subject to inconsistent and unfair evaluations. Such inconsistencies provide legitimate venues for criticizing an evaluation process and rendering its conclusions to be irrelevant. This presents the third important challenge in performance evaluation of public institutions. A useful approach must incorporate a mechanism that ensures a consistent application of judgment and evaluation of performance criteria. The evaluation process described in this paper is based on an amalgamation of tools traditionally utilized in scientific and engineering applications, primarily in control systems and pattern recognition (Bezdek, 1993). Fuzzy set theory and rule base systems are at the core of thousands of patents for instruments used to control everything from camcorders and small appliances to subway trains and ships. Although the application of these tools in areas of social sciences has been a much slower process (Treadwell, 1995), fuzzy rule based systems have been used to develop several successful applications in performance evaluation. These applications represent interdisciplinary team research collaborations supported by various development resources. Teams consisted of researchers in areas of management science, operations management, public finance and public administration, as well as various stakeholders in the particular applications. The remainder of this section includes brief descriptions of previously completed applications. The body of this paper describes the most recent research effort directed toward evaluating school districts in the state of New York. The first implementation is a system developed to evaluate state governments (for the 50 US states), specifically aspects related to financial management, capital management, and human resource management (Ammar, Wright, & Seldon, 2000; Selden, Jacobson, Ammar, & Wright, 2000). This application was later extended to evaluate the same aspects of governments of large US cities (Ammar, Dumcombe, & Wright, 2001; Ammar, Dumcombe, Hou, & Wright, 2001). These evaluations were part of the Government Performance Project (GPP) completed in 2000, by the Campbell Institute, Maxwell School of Citizenship and Public Affairs, of Syracuse University. The project was funded by the Pew Charitable Trust. The emphasis was on producing rankings of the fifty states in regard to each of the above aspects of management. These rankings were also compared to the rankings of journalists who performed independent evaluations using the same information provided by the GPP (Barrett & Greene, 1999). The applications address many of the challenges described earlier. Specifically, the objective was to develop an evaluation system that is fair and consistent. Moreover, the applications deal in detail with the challenges of comparing performances among similar

agencies where the evaluation included multiple and non-quantitative criteria. The second implementation is a process developed to evaluate the financial health of 35 large US cities. This was used to design a system that produces bond ratings and reports for the same cities, similar to those produced by rating agencies (Ammar, Dumcombe, Hou, Jump, & Wright, 2001). A subsequent application was developed; in collaboration with the public finance department of Fitch Ratings, to produce bond ratings and evaluation reports for 25 US airports. These applications deal with some of the same challenges described in the first implementation. In addition, they represent significant contributions as they provide evaluation systems that are transparent and can be clearly described. This is particularly important for credit rating agencies at a time when public attention is focused on concerns that these agencies have recently failed in their responsibility to provide consistent, reliable, and transparent ratings (US Securities and Exchange Commission, 2003). The third and most recent implementation is an evaluation of financial conditions of school districts in the state of New York. This project was funded by the state’s Education Finance Research Consortium. The evaluation criteria were designed with the help of an advisory board consisting of members from the state education department, school business officials, school board members, and other representatives of relevant government agencies. Affected by the state government financial crisis, school districts in the state of New York are currently facing difficult financial conditions. The aim of this project is to develop a financial condition indicator system (FCIS) geared toward identifying school districts that are on the verge of financial crisis in the short run, or those who are headed toward financial trouble in the long run. The project addresses many of the issues described earlier as well as introduces several new challenges. Developing a FCIS for the state of New York entails evaluating close to 700 school districts. Previous applications included no more than 50 evaluated agencies. This change in scope of the application introduced new challenges related to system development and operational complexities. Also, although all previous applications were developed with the help of an advisory group, the diversity in the composition and objectives of this particular group presents several additional challenges. Finally, the evaluation of school districts requires the inclusion of not only performance criteria but also of state regulations and requirements that in many ways limits the performance of the school districts. This is particularly challenging in cases where such regulations are not followed. This paper describes in detail the FCIS for evaluating school districts. It presents the process and challenges through which the system is developed. It also contains a description of the resulting outputs and reports. Section 2 includes motivation for the methodology and comparisons with other techniques. Section 3 contains the process of

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

developing the FCIS and a description of its various components. Section 4 includes a description of the model data and parameters. Section 5 and 6 contain descriptions of the output and assessments of the validity of the results. Section 7 includes final comments and concluding remarks.

2. Methodology Many attempts have been made to develop analytical models that could be used in evaluating performance of public institutions. Analytical techniques such as Data Envelopment Process, a linear-programming-based method, for assessing the comparative efficiency or productivity of organizational units, has been applied to a variety of contexts (Thanassoulis, 1999) including measurements of public sector efficiency (Ganley & Cubbin 1992). In this and in the previously mentioned applications, however, the focus is on evaluating performance as defined through an assumed domain of expertise rather than through the measurement of efficiency. In each example this proved to be a reasonable approach since experts with sufficient resources can describe reasonable conclusions about the financial health of the institutions. Issues arise with subjective expert evaluation when, as is almost always the case, there are limited resources (such as few experts or limited time.) Also, even when resources are available, such evaluations are subject to concerns about repeatable and verifiable consistency. These concerns are especially relevant when the result of one expert evaluation of an institution, is compared with the results for a different institution evaluated by the same expert or sometimes by a different expert. Systematic attempts at modeling expert judgment have of course been made in many contexts. Neural networks have been successfully used to develop intelligent systems. However, in most of these applications neural networks do not offer much potential for success. In general there have been too few evaluated entities and too many variables for training neural networks. For example, in one application there are only 25 airports in total being evaluated, based on 30 measures. In another, only 50 US States have their government financial management evaluated, using 25 measures. In the most recent application of evaluating school districts availability of data is not the issue. There is sufficient data to train a neural network with over 680 districts, using roughly 50 measures for each. The problem in this case, however, is that we have no overall performance measure on which to base the training. Getting experts to evaluate a substantial number of the districts in a consistent manner would be challenging and in effect impractical. Even if such a task could be completed we would be left with a black box that could perhaps rate the districts but could offer no insights as to how performance could be improved. Our objective remains that of developing an information rich system capable of evaluating numerous components

351

based on an existing domain of expertise. One method for evaluating multiple components is the Analytic Hierarchy Process (AHP) (Saaty, 1980). The AHP can be used as an evaluation model based on experts’ expressed preferences. However, in these applications, a defining feature of the evaluation process is the contextual contingencies within the evaluation. For example (as discussed in more detail in Section 3), debt levels for school districts are evaluated differently depending on the districts’ history of maintaining its physical plant. In order to incorporate contextual contingencies within AHP, experts will have to be presented with each of the contingent situations, as a combination of attribute values, for comparison. The number of comparisons that would have to performed by the experts quickly (exponentially) gets out of control. Another common paradigm for representing expert evaluation of multiple attributes is through the use of rule bases (Durkin, 1993). A rule base consists of a collection of IF –THEN statements expressing expert conclusions on given attributes. The ability to express conclusions as statements rather than relative comparisons facilitates the representation of contextual contingencies. These traditional rule bases have offered the best hope for modeling expert judgment but as discussed below provide challenges in complex applications as they may require an impractical numbers of rules.

3. Model development The development of the FCIS consists of employing a variety of analytical tools. This section includes the description of the model and the applied tools in a sequence that also outlines the rationale for the developed model. 3.1. Using knowledge based systems Knowledge based systems allow for the type of expert judgment required to fairly evaluate the financial condition of school districts. For example, one approach for evaluating the financial health of school districts is to consider the institution’s ability to manage debt. The district’s debt burden might be measured by considering annual debt service as a percent of total annual expenditures. In general a low debt service is preferred. However, for public institutions, such as school districts, debt service levels need to be evaluated in the context of the present condition of the district’s physical plant. Most capital improvement projects are funded by additional debt. An unwillingness or inability for a district to borrow money for this purpose can lead to future requirements for higher levels of expenditure in order to deal with badly deteriorating facilities. Hence in judging a district’s management of its debt, conventional wisdom suggests that you should also consider the district’s history of capital spending. Low current debt burdens combined with a poor

352

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

history of capital spending cannot be evaluated as having a good impact on the district’s financial condition. On the other hand a moderately high debt burden combined with a good level of historical spending on capital improvements could be evaluated positively. A knowledge based system can represent these judgments in terms of rules. One rule may state that if a district has a low debt burden and a poor history of capital spending then the judgment regarding debt management is fair. In our application capital spending is measured in dollars spent per students (those dollars are adjusted for inflation and regional cost indices). A simple average over a period of several years can be used as a measure of historical commitment to capital projects. But at what point is that spending level considered ‘low’? By looking at the spending levels per student for some 600 districts a sense of what is relatively low and high is obvious but the exact point at which a spending level is identified as low or not is more difficult to determine. And even if a cutoff point it is determined, are we prepared to accept the fact that at some point spending one additional dollar will result in a district moving from a low spending level to one that is high? A possible solution to this abrupt jump from low to high is to have more gradations of level of capital spending per student. We could produce maybe seven levels of spending described as ‘very low’, ‘low’, ‘moderately low’, and so on up to ‘very high’. However, the same issue would apply to measuring the debt burden and we need perhaps seven gradations for this measurement as well. As a consequence we will have created 49 possible combinations and the need for as many rules to make judgments about all possible capital spending and debt service levels. Although this lessens the impact, we are still left with the fact that one more dollar can result in a jump from one rule to another and a corresponding jump in the evaluation. In reality debt management also needs to be evaluated in the context of the extent to which the debt level is approaching statutory or constitutional limits. This adds yet another dimension of complexity in the evaluation. If the outstanding debt level is approaching the legally imposed limit then concerns are raised regarding the district’s ability to handle unexpected physical plant problems that require additional capital spending, particularly if there is a history of less than adequate capital spending. Hence if we add the percent of the debt limit reached as a third factor in the rule base, and again argue that we need something like seven gradations for that measure, we will have increased the number of rules relating to debt management to 73 or 343. In knowledge based systems these rules are typically created by or extracted from a group of experts. Suggesting that these experts need to create conclusions for these hundreds of rules seems daunting already. The unfortunate reality is that we have not even begun to evaluate debt management alone, much less the overall financial health of a school district. In fact, the eventual model described in this paper includes close to fifty measures. If we considered seven

gradations in each, that would require us to define 750 rules, which of course is absurd. The large number of rules necessary for complex applications, such as the comprehensive evaluation of the financial health of public institutions, makes the use of crisp knowledge based systems impractical. It is for that reason that in developing our model we consider applying fuzzy set theory within the representation of the knowledge base systems. 3.2. Using fuzzy rule based systems Fuzzy set theory provides us with the opportunity to model an evaluation of debt management (with all its complexity) using far fewer rules. Fuzzy sets incorporate an infinite number of gradations in categories through its membership function defined in [0,1]. For example three broad categories (or fuzzy sets) for debt service can be defined, namely high, moderate, or low (see Fig. 1.) Memberships are defined as trapezoidal functions ð1½0; 1Þ over possible debt service values (horizontal axis.) The overlap in sets indicates the possibility for a value to have simultaneous membership in more than one set. This eliminates potential anomalies resulting from abrupt transitions where one more dollar might cause the debt service to jump into a different category. A district can have a debt service that is high to some extent while at the same time being moderate to a degree as well. As an illustration (see Fig. 1), an annual debt service that is 4.9% of total annual expenditures would have a membership of 0.39 in the high set and at the same time have a membership of 0.61 in the moderate set. This implies that applicable evaluation rules are those related to a high debt burden as well as those related to a moderate debt burden. Creating categories of fuzzy sets and specifying the transition between sets are part of the system parameterization that is discussed in a later section. Similarly, three categories (fuzzy sets defined by trapezoidal functions) are specified for the other two factors described earlier as impacting judgment on debt management. These are the level of historical capital spending and debt service as a percent of the regulated debt limit. Since each of the three factors is represented using only three categories (low, moderate, and high) there are 27 rules that need to be defined to represent judgment on all the combinations of the three factors. Fig. 2 contains a rule base representation in a matrix form. The matrix has three dimensions representing each of the factors of debt management (debt burden, capital spending, and % of debt limit.) Within each dimension the three categories (low, moderate, and high) are listed. The matrix hence includes 27 cells each representing a rule. The entry in the cell (good, fair, or poor) is the judgment corresponding to the three-factor combination. In Fig. 2, the shaded cells represent the rules that are applicable for a selected school district. This particular school district has a debt burden that is high to moderate; capital spending that is high to

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

353

Fig. 1. Fuzzy set categories for debt service.

moderate; and a % of debt limit that is low to moderate. Therefore eight rules are applicable to the evaluation of the debt management of this school district. Three of the rules lead to a poor conclusion, four lead to a fair conclusion and only one leads to a good conclusion. To illustrate the matrix notation consider the rule that leads to the good conclusion. The same rule can be stated as follows:

IF

the debt burden is moderate AND the % of the debt limit is low AND the capital spending is high THEN the evaluation of the debt management is good. For the purpose of efficiency and to insure consistency in the rule application, adjacent rules in the matrix that lead to

Fig. 2. Rule base matrix.

354

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

the same conclusion are combined using logical operations. Therefore the 27 rules are further reduced. For example, consider the two shaded rules in the middle block of the matrix. The two rules lead to the same conclusion of fair. The two rules are considered adjacent in that they differ only in one dimension, namely the level of capital spending. Hence the rule conclusion is the same regardless of whether the capital spending is moderate or high. The logical combination of the two rules describes a level of capital spending that is NOT low. The two rules can be combined to read as follows: IF

the debt burden is moderate AND the % of the debt limit is moderate AND the capital spending is NOT low THEN the evaluation of the debt management is fair. An algorithm completes this rule combining process automatically in all dimensions of the matrix. For a detailed discussion of the rationale behind rule combination we refer the reader to Ammar, Duncombe, Bifulco, & Wright (1999). The evaluation of debt management for each school district is completed by applying all identified rules. Rule application is performed using the generally established Extension Principle (Dubois & Prade, 1988). Based on the memberships of the evaluated factors, the extension principle defines degrees of conclusion for the relevant rules. As a result of fuzzy theoretic rule application, the evaluation of the debt management for the selected district is determined to be poor to a degree (0.39), largely fair (0.61), and even good to a limited degree (0.06) (see Fig. 2.) Slight changes in any of the factors would result in only slight changes in the degrees of these conclusions. For that reason, fuzzy rule based systems effectively model expert judgment with a manageable number of rules and without concern that small variations in evaluated factors would create jumps in the overall assessment. For a detailed discussion of fuzzy theoretic rule application we refer the reader to Ammar, Dumcombe, & Wright (2001). 3.3. Using a multi-level fuzzy rule based system Debt management is only one of many important factors in evaluating the financial condition of school districts. Many factors can be identified from a variety of perspectives. Much of the literature related to financial health or condition focuses on specific components of fiscal health. Concern with immediate fiscal crises has lead to an emphasis on various standard financial ratios to measure ‘cash solvency’ or ‘budgetary solvency’ (Nollenberger, 2003). These measures have focused particularly on general fund balances (Dearborn, 1988; Gold, 1986). While surplus and balance measures can be valuable fiscal indicators, they are not perfect measures of a government’s fiscal health. For this reason there has been significant research on

the ‘underlying’ or ‘structural’ fiscal health of a government. This research generally has divided the analysis of fiscal health into two parts: the capacity to raise taxes (fiscal capacity) and the level of expenditures required to provide a standard package of public services of average quality (expenditure needs) (Ladd & Yinger, 1989). Another perspective on the fiscal health of state and local governments comes from the credit rating agencies. Their focus is on the ability to service debt obligations. Fitch, Moody’s, and Standard & Poor’s all consider economic, fiscal, debt, and administrative factors, and they identify a core of ten to twenty variables that receive attention (Fitch, 2000; Moody’s Investors Service, 1999; Standard & Poor’s, 2000). Some authors have taken a broader view of financial condition that brings many of these factors together (Berne & Schramm, 1986; Mead, 2001; Nollenberger, 2003; Office of the State Comptroller, 2002). Measures of fiscal health have often been grouped by whether they measure short-run fiscal health or long-term fiscal health, and by the object of the measures. For example, the ICMA Financial Trend Monitoring System (Nollenberger, 2003) includes measures of cash solvency, budgetary solvency, long-run solvency, and service-level solvency. Mead (2001) has organized financial indicators into measures of liquidity, financial position, solvency, fiscal capacity, risk and exposure, and economic base. Berne and Schramm (1986) develop a framework for measuring financial condition that compares expenditure pressures (from present service demands and past commitments) to available resources (from either internal reserves or external revenues and grants). Shortterm measures of liquidity, fund balances, and budget surpluses/deficits are reflected in past expenditure commitments and available internal resources. Current expenditure pressures and available external resources include measures of the long-term fiscal health of the government and the impact of past financial decisions. The overall model described in this paper builds on the research briefly discussed above to develop a broad-based system to measure the financial condition of school districts. However, instead of trying to combine all the relevant factors into one large fuzzy rule base, the evaluation is decomposed into smaller segments. The decomposition makes the system development manageable and allows evaluation of smaller subcomponents. The decomposition also suggests a hierarchy in the subcomponents of the evaluation. For example the school tax rate aspect of the evaluation of a district can be assessed based on four factors: tax dollars as a percent of property value, tax dollars as a percent of income, property value per student, and income per student. All four factors measure the contribution of the school tax rate to the financial conditions of the district. As described earlier, a fuzzy rule base system with these four factors can be designed to evaluate the tax rate aspect. Also, as aforementioned, each district can be evaluated in the fuzzy set theoretic sense. The rule

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

application leads to conclusions with respective degrees for the evaluation of good, fair, and poor. This fuzzy output could serve as input factors to the component presented at the level above in the hierarchy. Specifically, the school tax rate evaluation could be considered as one of the components used to evaluate the overall ability to raise taxes. The latter, therefore, is above the former in the hierarchy. Evaluating the ability to raise taxes may also include measures for the trend in tax rate increases, the extent to which budgets have been defeated in the past, and a measure of how closely assessed values compare to taxed values. The hierarchy may continue, in that the fuzzy output resulting from the evaluation of the ability to raise taxes could be then combined with fuzzy measure for state aid, diversity of revenue, and stability of revenues, in a higher rule base that would evaluate the district’s ability to manage revenues. In turn revenue management can be a component of a higher rule base evaluating long run financial condition of the school district. Similarly decomposed evaluations can be made to describe a hierarchy of evaluating the short term financial condition of a school district, or the economic conditions faced by the school districts. The decomposition of all the factors into separate rule bases is presented in Figs. 3 and 4. (Note that student

355

performance is determined to play a role in the evaluation of the financial condition of a district. For example, poor student performance could place high demands on future resources as districts attempt to meet federal and state performance requirements). In total a hierarchy of 21 rule bases is used to evaluate the financial conditions of a school district with a total of 49 different measures. The process of describing a structure for evaluation, like that given in Figs. 3 and 4, is crucial to designing a successful application. In many ways it is an art that requires an overall understanding of the evaluation process and the interdependencies among factors. It also requires an understanding of the evaluation judgments as they relate to the tradeoffs among performance measures.

4. Model parameters The implementation of the FCIS requires the identification of appropriate data and specification of several parameters. These include parameters describing the evaluation criteria in the form of expert rules, and parameters describing the categorization of performance measures in the form of fuzzy sets. This section contains the process of data selection and parameter determination.

Fig. 3. Overall evaluation hierarchy.

356

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

Fig. 4. Economic evaluation hierarchy.

4.1. Collecting data The financial condition indicator system (FCIS) was initially designed for school districts in New York. However, the goal is to illustrate a system that can be implemented with readily available data in any state. Hence the system was designed to utilize available data from the State Education Department, New York Department of Labor, Office of the State Comptroller, and US Bureau of the Census. The financial information used in FCIS is based on unaudited annual financial statements submitted by school districts, commonly labeled as ST3 reports. The latest financial information available at the time of system development was for the 2000– 2001 fiscal year. 4.2. Creating categories of performance measures The initial list of the evaluation measures was selected by the researchers with expertise in public finance and incorporates the recent work in finance and public education. In order to provide a level of expertise that reflects the judgment of school finance professionals, an advisory board was created. The members included representatives from the Office of Audit Services in the State Education Department and the Office of the State Comptroller, a representative of the New York Association

of School Business Officials, a CPA with expertise in auditing school districts, school district administrators and school board members. Based on the advisory board’s recommendations a number of different measures were considered before the final set was selected. Tables 1– 3 contain descriptions of these final measures. After the measures were selected they were grouped into a hierarchy of factors for evaluation rule bases as shown in Figs. 3 and 4. Ideally each rule base will have at most four factors in order to keep the number of rules at a manageable level. Any given rule base will have no more than 81 or 34 rules (since each factor is described using three categories). For example, as shown in Fig. 3, the fund balance rule base uses four factors: the current unreserved fund balance (both unappropriated and appropriated, UUB þ UAB), the unreserved unappropriated fund balance (UUB), the trend in the unreserved balance, and the reserved fund balance. All balances are expressed as a percent of annual expenditures. These factors were chosen because each provides a different measure of the impact of fund balances on financial stability and because the interactions between them present a complete picture of the evaluation. For each factor three categories (fuzzy sets) are defined and trapezoidal membership functions (as illustrated in Fig. 1) are created. For the unreserved fund balance these may be described as low, moderate and high. The trapezoidal

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364 Table 1 Short-run financial condition measures Factors

Table 2 Long-run financial condition measures Measure

Source

Liquidity General fund quick ratio (assets including cash, receivables, ST investments divided by current liabilities) Multiple funds quick ratio (general fund, special aid fund, food service fund) Fund balances as a percent of total expenditures: (general fund, special aid fund, food service fund) Unreserved, unappropriated fund balance (UUB) Unreserved fund balance (appropriated þ unappropriated)

Reserved fund balance

2001 average (1998–2000)

2001

Average (1999–2001) 2001 trend from 1997– 2001 (weights later years heavier) 2001 or average (1999–2001)a

ST3

ST3

ST3 ST3

Factors

Measure

Source

2001

NY Comp

2001

NY Comp, SED

2001

NY Comp

1992–2001

NY Comp

Debt limit: percent of debt limit used

2001

NY Comp, SED

Capital spending per pupil adjusted for regional cost differences and inflation

Average (1999–2001)

NY Comp, SED

Average (1991–2001)

NY Comp, SED

2001

NY Comp

2000

NY Comp, SED

2001 2000 1996–2001

NY Comp SED NY Comp

2001

NY Comp

1997–2001

SED

Local revenue diversification: property tax as percent of total local revenue

Average (1999–2001)

NY Comp

Revenue stability: average variation around the regression line

1991–2001

NY Comp

Aid dependency: state aid and federal aid as percent of total revenue

Average (1999–2001)

NY Comp

Debt measures Debt ratios Long-term debt outstanding relative to property values Long-term debt multiplied by building aid ratio relative to property values Adjusted debt service as percent of total expenditures Percent of debt paid off over 10 years

ST3

Tax capacity measures Market property values per pupil in 2001 Property tax burden (property taxes/property values) Number of budget defeats in last five years

357

2001

NYComp

2001 trend from 1998– 2001 1997– 2001

ST3/NYComp SED

Source: ‘ST3’, annual financial statements submitted by school districts; ‘SED’, NY State Education Department; ‘NYComp’, New York State Office of the State Comptroller. a Minimum of the average from 1999 to 2001 or the values in 2001.

functions can be created by indicating the range in, which a balance can be both low and moderate, and the range in, which it can be both moderate and high. These ranges are known as fuzzy set transition points. The ranges can be defined by observing the distribution of the data or by using judgments derived from the experts. Extracting transition of categories through the data distribution can be achieved by considering various percentiles. For example, the 10th, 50th and 90th percentiles for the unreserved fund balances (UUB þ UAB) are 1.0, 5.4, and 12.9%. As shown in Fig. 5, the UUB þ UAB is described as being low for values below 2%, low and moderate between 2 and 5.5%, moderate and high between 5.5 and 13.0%, and fully high above 13%. These transition points roughly match the stated percentiles and were acceptable parameterizations by the advisory board. The corresponding percentiles for the second factor, namely the unreserved unappropriated balance (UUB), are 1, 2.3, and 6.5%. In this case the advisory board strongly argued against basing the membership functions on the levels derived from the data. New York State regulations require that a school district’s UUB must not exceed 2% of the district’s expenditure. Hence while over half the districts

Revenues Property taxes Tax burden Property taxes relative to property values Property taxes relative to income Property values per pupil Income (AGI) per pupil Trend in tax burden relative to property values Assessment ratio (assessed value/full value) Budget referendum defeats in last 5 years

Source: ‘SED’, NY State Education Department; ‘NYComp’, New York Office of the State Comptroller.

appear to ignore that regulation, it seems inappropriate to define the low category to include a fund balance that is higher than the allowed limit. For that reason, the categories for the UUB are not based on data percentiles but rather are defined using more appropriate transition points of 0.0, 1.5 and 2% (see Fig. 5). Similarly, the transition points for the reserved fund reflect the experts’ judgment that there is a range (between 3 and 6%) in which the fund balance should be regarded as fully moderate. The process of creating

358

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

Table 3 Economic condition measures Factors

Measure

Source

Average (1999–2001) Average (1999–2001)

SED

Average (1999–2001)

SED

Regional cost index

1998

SED

Sparsity Pupils per square mile Districts with population below 500 students

2001 2001

SED SED

2000 trend (1995–2000)

SED

2001 trend (1996–2001)

SED

1990–2000 2000 trend (1990–2000)

Census Census/SED

1991–2001 1996–2001

SED SED

1991–2001 and 1996–2001

SED

1996–2000 2000–2002

NYDOL NYDOL

Average (2000–2002) Average (1998–2000)

NYDOL

Cost factors Student needs Share of K6 students getting free lunch Share of K12 students classified as limited English proficient Share of K12 students classified as high cost special needs

Fiscal capacity Income (AGI) per pupil Market value of property per pupil Population/Enrollment Population growth Pupils per capita Enrollment Growth Stability: average variation around the regression line Employment (county-level): Employment growth rate Unemployment rates High wage employment share (manufacturing, TPU, FIRE)

SED

NYDOL

Source: ‘Census’, US Bureau of Census; ‘SED’, NY State Education Department; ‘NYDOL’, New York Department of Labor.

categories for performance factors is an iterative process and involves in depth consultation with the advisory board.

is defined, consider the fund balance rule base given in Fig. 6. The experts must specify the rule conclusions for all 81 combinations of the three categories for each of the four evaluation factors. The rule matrix assures that rule base is complete and that all the possible combinations are considered. It also allows the expert to check for consistency in the rule conclusion. It is easy to see that similar rules have similar conclusions, and that in each dimension as factors improve the conclusions reflect these improvements. Usually cells that represent clear judgments are the first completed rules. Subsequently adjacent cells are filled in to complete the matrix in a consistent manner. This process of completing these rule matrices is also iterative. Once a set of rules has been created they are tested using districts in which the experts have some prior judgments. The system developed allows the user to evaluate the rule base for any district and to see not only the result but the group of rules that contribute to that result. Fig. 6 shows 16 shaded cells that apply in the evaluation of the fund balance of a selected district. Four rules lead to a poor conclusion, ten to a fair conclusion, and two to a good conclusion. Any rule can be modified by merely changing the conclusion in the appropriate cell (changing a, g to an f would change the conclusion from good to fair.) The impact of the changes can be evaluated by re-executing the rule application calculations. In addition, as the experts work to create a complete set of rules it is often helpful to know if a particular rule will actually apply for any of the evaluations. The system includes a counter for each rule that determines the number of districts whose measures of performance would qualify the rule to be applicable (see Fig. 7). This feature facilitates the iterative process of rule definition. The counter is also helpful in selecting the transition points for the membership functions. For example, it is important to choose fuzzy set transitions that would provide enough distinction among the districts and not have the majority of districts evaluated with a small section of the rule base. All 21 rule bases are similarly created. As the user moves up through the structure, fuzzy output from one rule base becomes input to higher rule bases. The final evaluation is now based on the interaction between the long run and short run financial conditions, the economic environment, and student performance. The four factors are combined in the overall evaluation rule base.

5. Model output 4.3. Creating rules Once the fuzzy sets are defined for each of the measures and a general understanding of the category transitions is reached, the rules can be articulated accordingly. The mechanism for identifying the rule base is through identifying matrices such as that illustrated in Fig. 2. For the sake of further illustrating the process by which a rule base

The most basic model output is the result of the overall rule base, in the form of a conclusion specifying degrees associated with poor, fair, and good performance. These values can be defuzzified (Dubois & Prade, 1988) to a single number that can be used to rank districts statewide or within a selected group of peer districts. However, the advisory board favors reporting only the fuzzy measures

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

359

Fig. 5. Fuzzy set categories for factors of fund balance.

believing they provide a more descriptive evaluation while at the same time avoiding direct district to district comparisons. In addition to obtaining the results of the final rule base, the fuzzy measures of lower level rule bases can also be of value. For example, district administrators might like to investigate how they are performing outside of the context of the economic environment. A look at the results for the short run and long run financial condition rule bases could provide those insights.

Potential model outputs include a detailed report of the result of every rule base as well as all model inputs. For example, Fig. 8 includes the results of the overall evaluation and the four major contributing evaluations for a sample district, district A (Note that the student performance factor is a single measure based on Math and English exams in the fourth, eighth and high school grades. The index ranges from 0 to 200; for district A the measure was approximately 160). Fig. 9 gives the details of the short

Fig. 6. Fund balance rule base.

360

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

Fig. 7. Rule counter.

run component of the overall evaluation. It is an illustration of how all system component information and evaluation can be presented. This detailed output allows district officials to see where their evaluation is poor and where it is good as well as what components may be driving the overall evaluation. Section 6 includes an example of how this output can specifically be interpreted.

6. Assessing the validity of the model In order to assess the validity of the model, members of the advisory board were asked to identify a set of

school districts whose financial conditions they believe would be particularly weak, as well as a set they believe would be strong. Collectively they identified 32 strong districts and 20 weak ones. The weak districts included those identified by the State Education Department as potential problems based primarily on the fund balances. The strong districts were for the most part identified based on specific knowledge of individual board members. However, both lists are intended to be illustrative and not exhaustive. Since a primary reason for developing the FCIS is to create a warning system for schools that are in financial trouble, most of our attention is focused on the weak list.

Fig. 8. Overall evaluation of sample district.

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

361

Fig. 9. Short run evaluation of sample district.

The basic model outputs for the 20 weak districts are shown in Table 4 (New York State Education Department has requested that districts not be identified by name). It includes defuzzified scores that can be used to compare or rank institutions. A score of 0 represents fully poor, 10 fully fair, and 20 fully good. The outputs preferred by the advisory board are in the form of fuzzy degrees associated with poor, fair and good performances. These are included in Table 4 for the overall evaluation as well as the top four factors (Detailed output of all factors is available as described in Section 4). Of the 20 districts identified as weak, 17 receive evaluations in which the poor degrees exceeds the fair degrees resulting in defuzzified scores of less than five. Nine of the districts are evaluated as fully poor. In fact out of all 683 districts there are only these 9 that received this minimum mark. The remaining three on the weak list receive evaluations that are poor, fair, and good to some degree. In one case the poor exceeded the good resulting in a rating of less than fully fair (defuzzified score of 8). In the other two cases the good exceeded the poor resulting in a rating higher than fully fair (defuzzified scores of 12 and 13). Each of the three cases was presented to the advisory board along with detailed explanations of how the conclusions had been reached. For example, District #6, which has

the highest defuzzified score among districts identified as weak, is judged to be good with a degree of 0.38 and fair with a degree of 0.62 (and a small degree of poor). The entire picture for District #6 could be investigated by looking at the detailed model output described in Section 5. In fact, District A is actually our District #6. Another effective way to understand the overall evaluation of a district is to start at the top result and look at the rules that lead to the final outcome. For each rule base the system tracks the critical rules or the rules that have the biggest impact on the evaluation. These results for District #6 are shown in Fig. 10. The top half of the figure contains the fuzzy results for the four factors leading to the overall evaluation. The lower half contains a description of the critical rules that determine the extent to which the district is evaluated as poor (0.09), fair (0.62), and good (0.38), using those four factors. For example the good conclusion results from the extent to which the short run is good, the long run is NOT poor, the performance is NOT poor, and the economy is good. Together the critical rules indicate that the short run analysis is crucial to the overall evaluation. Understanding the short run analysis can be enhanced by looking at the same type of output for that rule base. Fig. 11 contains the results of the short run rule base for District #6. The fuzzy measures for the contributing factors

362

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

Table 4 Overall evaluation for districts indentified by advisory board as weak District #

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Defuzzified score

3 12 0 8 4 13 5 1 2 0 0 5 0 4 0 0 2 0 0 0

Overall

Short term

Long term

Performance

Economic

Poor

Fair

Good

Poor

Fair

Good

Poor

Fair

Good

Low

Mod

High

Poor

Fair

Good

0.76 0.14 1.00 0.28 0.62 0.09 0.73 0.91 0.85 1.00 1.00 0.56 1.00 0.63 1.00 1.00 0.85 1.00 0.99 0.99

0.24 0.66 0.00 0.72 0.38 0.62 0.27 0.09 0.15 0.00 0.00 0.44 0.00 0.37 0.00 0.00 0.15 0.00 0.01 0.01

0.00 0.34 0.00 0.10 0.00 0.38 0.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 0.00 1.00 0.19 0.92 0.00 0.73 1.00 0.85 1.00 1.00 0.88 1.00 1.00 1.00 1.00 0.92 1.00 0.99 1.00

0.00 0.13 0.00 0.81 0.08 0.62 0.27 0.00 0.15 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.08 0.00 0.01 0.00

0.00 0.87 0.00 0.00 0.00 0.38 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.57 0.32 0.69 0.45 0.62 0.21 0.81 0.70 0.92 0.68 1.00 0.90 1.00 0.84 1.00 0.79 0.71 0.83 0.83 0.70

0.43 0.37 0.31 0.55 0.38 0.79 0.19 0.30 0.08 0.32 0.00 0.10 0.00 0.16 0.00 0.21 0.29 0.17 0.17 0.30

0.00 0.63 0.00 0.39 0.00 0.03 0.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.14 1.00 0.00 0.00 0.00 0.00 0.00 0.64 0.04 0.00 0.00 0.05 0.00 1.00 1.00 0.00 1.00 0.00 0.00

1.00 0.86 0.00 1.00 1.00 1.00 1.00 1.00 0.36 0.96 1.00 0.56 0.95 0.63 0.00 0.00 1.00 0.00 1.00 1.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 0.00 0.37 0.00 0.00 0.00 0.00 0.00 0.00

0.52 0.66 1.00 0.28 0.18 0.09 0.27 0.47 0.25 0.41 0.52 0.14 1.00 0.00 0.59 0.92 0.71 0.63 0.52 0.55

0.48 0.34 0.00 0.72 0.55 0.38 0.73 0.53 0.75 0.59 0.48 0.27 0.00 0.30 0.41 0.08 0.29 0.37 0.48 0.45

0.24 0.13 0.00 0.10 0.45 0.62 0.20 0.09 0.08 0.00 0.10 0.73 0.00 0.70 0.00 0.00 0.15 0.00 0.01 0.01

indicate that while the fund balance was largely poor, the tax capacity is good and the liquidity is fair to good. The critical rules indicate that the good evaluation of the tax capacity and fair to good liquidity greatly influences the short run evaluation. The model user can continue to the next level and look at the detail of the liquidity, tax capacity, or fund balance rule base, if desired. Eventually a picture emerges of a district that has been keeping very low fund balances for years (although not much below the State’s maximum for unreserved funds) but has maintained a reasonable level of liquidity and has the ability to raise taxes substantially if necessary. Taxes have been declining over the past three years and the tax burden is actually low in a more than moderately

wealthy community. The overall economic picture is healthy and student performance is not deficient. One can very easily argue that this is a district that has been well managed financially and one that has been able to deliver reasonable performance without relying on high levels of reserves. The advisory board fully agreed with this evaluation. Actually they were delighted to see that the model could distinguish between districts that had low fund balances because they were in financial trouble from those that could survive with low balances because they were well managed. The board also concluded that the FCIS model also better represented the other two districts labeled as weak. At the other extreme, three of the list of 32 strong districts received

Fig. 10. Critical rules in overall evaluation of district #6.

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

363

Fig. 11. Critical rules in short run evaluation of district #6.

evaluations from the model that were more poor than good. Again, in these cases, the advisory board endorsed the model’s assessment. In general the board was impressed at the system’s ability to model the kinds of judgments knowledgeable humans would make.

7. Concluding remarks The Financial Condition Indicator System (FCIS) described earlier was developed as a pilot project for the New York State Education Department with funding provided by the state’s Education Finance Research Consortium. The next phases for the project are currently under discussion. Some members of the advisory board see great potential for using the model to enable school business officers to better understand the critical factors in managing their district’s finances. One of the authors has been conducting training sessions around the state using model outputs as the basis for discussions with the business officers. Other members of the advisory board see benefits arising from state officials using the model as an early warning system for districts that might be getting into financial trouble. Another group sees the potential to develop the system even further to better respond to the No Child Left Behind Act. This act requires various levels of public reporting of school district data, including financial data. Many school officials are concerned that publicly available raw data presented without context could create unnecessary confusion and misinformation. For example, let’s return to debt burdens described earlier in the paper. A district with slightly above average debt levels combined with a history of carefully managed capital spending should be evaluated more favorably than a district with low debt levels but no history of reasonable capital spending. Any publicly available report that merely lists debt burdens might lead some parents to assume that the district with the lower debt levels is performing better than the district with the higher levels.

The output of the fuzzy rule base system could be used to produce automated reports for all districts that included the relevant context (based on the critical rules) for each evaluation. From the varying perspectives of the advisory board, the Financial Condition Indicator System has the potential to be used to evaluate districts, to identify potential problems before they become unmanageable, to advise district officials on how to improve their financial management, and to help responsibly keep the public informed about the financial management of their schools.

References Ammar, S., Duncombe, W., Bifulco, R., & Wright, R. H. (1999). Consistency in fuzzy rule-based systems: an application in elementary education performance evaluation. Proceedings of 18th International Conference of the North American Fuzzy Information Processing Society, New York, NY, pp. 120–124. Ammar, S., Dumcombe, W., & Wright, R. (2001). Evaluating capital management: a new approach. Public Budgeting and Finance, 21(4), 47– 69. Ammar, S., Dumcombe, W., Hou, Y., & Wright, R. (2001). Evaluating city financial management using FRBS. Public Budgeting and Finance, 21(4), 70–90. Ammar, S., Dumcombe, W., Hou, Y., Jump, B., & Wright, R. (2001). Using fuzzy rule-based systems to evaluate overall financial performance of governments: an enhancement to the bond rating process. Public Budgeting and Finance, 21(4), 91–110. Ammar, S., Wright, R., & Selden, S. (2000). Ranking state financial management: a multi-level fuzzy rule based system. Decision Sciences, 31(2), 449– 481. Barrett, K., & Greene, R. (1999). Grading the states. Governing, 12, 17– 90. Berne, R., & Schramm, R. (1986). The financial analysis of governments. Englewood Cliffs, NJ: Prentice-Hall. Bezdek, J. C. (1993). Fuzzy models—what are they, and why? IEEE Transactions on Fuzzy Systems, 1, 1– 5. Dearborn, P. (1988). Fiscal conditions in large American cities, 1971–1984. In M. McGeary, et al. (Eds.), Urban Change and Poverty. Washington DC: National Academy Press. Dubois, D., & Prade, H. (1988). Possibility theory. New York, NY: Plenum Press.

364

S. Ammar et al. / Expert Systems with Applications 27 (2004) 349–364

Durkin, J. (1993). Expert systems: catalog of applications. Akron, OH: Intelligent Computer Systems Inc. Fitch Ratings, (2000). Local government general obligation rating guidelines (May 23). Tax supported special report, pp. 1 –11. Fried, H. O., Lovell, C. K., & Schmidt, S. S. (1993). The measurement of productive efficiency. New York, NY: Oxford University Press. Ganley, J. A., & Cubbin, J. S. (1992). Public sector efficiency measurement. Amsterdam: North-Holland. Gold, S. (1986). State government fund balances, financial assets, and measures of budget surplus (vol. 2). in Federal-state-local fiscal relations, Technical Papers, Washington DC: US Department of the Treasury, p. 596. Ladd, H., & Yinger, J. (1989). America’s ailing cities: fiscal health and the design of urban policy. Baltimore: The Johns Hopkins University Press. Mead, D. (2001). Assessing the financial condition of public school districts. Selected papers in school finance, 2000–2001. Washington, DC: National Center for Education Statistics. Moody’s Investors Service, (1999). The determinants of credit quality. Municipal Credit Research, November, 1–16. Nollenberger, K. (2003). Evaluating financial condition: a handbook for local government. Washington DC: ICMA.

Office of the State Comptroller, (2002). Financial condition analysis. Local government management guide, Albany, NY: Office of the State Comptroller. Rossi, P. H., & Freeman, H. E. (1993). Evaluation: a systematic approach. Newbury Park, CA: SAGE. Saaty, T. L. (1980). The analytic hierarchy process. New York, NY: McGraw-Hill. Selden, S., Jacobson, W., Ammar, S., & Wright, R. (2000). A new approach to assessing performance of state human resource management systems: a multi-level fuzzy rule-based system. Review of Public Personnel Administration, 20(3). Standard and Poor’s, (2000). Public finance criteria. New York: Standard and Poor’s. Thanassoulis, E. (1999). Data envelopment analysis and its use in banking. Interfaces, 29, 1–13. Treadwell, W. A. (1995). Fuzzy set theory movement in the social sciences. Public Administration Review, 55, 91– 98. US Securities and Exchange Commission, (2003). Report on the role and function of credit rating agencies in the operation of the securities markets. Washington DC: US Securities and Exchange Commission.