- Email: [email protected]
Transit performance evaluation in the U.S.A.
Transpn. Res.-A, Vol. 26A, No. 6, pp. 483-491, 1992 Printed in Great Britain.
0191-2607/92 S5.00 + .00 © 1992 Pergamon Press Ltd.
T R A N S I T P E R F O R M A N C E E V A L U A T I O N IN THE U.S.A. GORDON J. FIELDING University of California, School of Social Sciences, Irvine, CA 92717, U.S.A. Abstract-Performance of transit agencies in the United States improved during the 1980s. At the beginning of the decade, Americans had become disenchanted with transit; legislation was passed that required agencies to report performance and accept regular audits. Theory underlying these policies is examined in four components: dimensions for policy objectives, indicators, information systems and incentives. Three programs are examined: federal triennial reviews that monitor compliance with planning and grant requirements, California performance audits that analyze goals and track performance on five indicators and the Los Angeles program that encourages improvement by offering incentive payments for better-than-average performance. The California audits have been the most successful.
INTRODUCTION
Improving public transport is an art; we are not sure what works, because systematic evaluation of performance against intended goals has not been conducted for metropolitan areas. Studies have focused on single modes, overlooked the goal of increasing highway capacity, and emphasized efficiency when efficiency was not mentioned as a goal for transit until the mid 1970s. Even the evaluations mandated by federal, state and local legislation in the United States are deficient: objectives are not adequately defined and measurement criteria do not indicate standards against which performance might be assessed. Evaluation of public transit is made difficult by changing goals. Preserving the commercial advantages of central cities and the jobs of transit employees was initially important; then social, environmental and conservation goals were added in the 1970s (Fielding, 1987). Federal investment in public transit began when older, central cities became alarmed over the rate of federal investment in highways and the abandonment of commuter service by railroads (Smerk, 1991). To this were added a whole host of social and environmental goals during the 1970s that transit alone was incapable of achieving. As management has struggled to respond to these changing goals, performance of transit has declined. Public transit in the U.S., which had been covering operating costs from the fare box until 1965, was recovering only 54070 in 1975 and 42070 in 1985. For the fiscal year ending in 1989, the annual operating deficit was $7.1 billion, 48070 of costs. And transit ridership was virtually unchanged despite the generosity of federal, state and local governments. Transit had reversed the downward trend of ridership and cost recovery, and had maintained its share of travel to work, but all this had been achieved at a very high cost per passenger. Alarm over operating deficits caused legislators to seek different solutions. Performance reviews were mandated in the early 1980s as a way to force
some discipline into public transit while continuing the earlier expectations. As a result, cost increases have been controlled and operating revenue increased: the ratio of operating revenue to cost has increased from 38°7o in 1980 to 42070 in 1985, and 48070 in 1989; real cost increases in the last half of the 1980s were less than 1070per annum compared to increases of between 4 and 507o over inflation between 1970 and 1985 (U.S. DOT, 1991). Performance auditing cannot claim exclusive credit for cost control, but it has focused attention on the unbridled expenditures of the 1970s and the need for improved performance in every agency. Three performance programs will be reviewed: the federal requirement for triennial audits, the California performance audits for operators receiving assistance from the Transportation Development Act and the performance incentive program utilized by the Los Angeles Country Transportation Commission. Each will be judged in terms of intentions, techniques of measurement, and use of results to improve performance. PERFORMANCE THEORY
Performance evaluation of public enterprise is based on the premise that administration needs to be untangled; that a small set of indicators can be used to distinguish "good" from "bad" performance. Indicators provide a "bottom line" that, although not as generally accepted as those used by private enterprise, nevertheless provide measures helpful when gauging success or failure. Systems for performance evaluation require four components: 1. dimensions that represent the objectives that motivated public intervention; 2. indicators that translate objectives into quantitative measures; 3. an information system that gathers appropriate data in a consistent manner to provide crosssectional and time-series statistics; and
483
484
G, J. FIELDING
4. an incentive system that rewards managers for improving performance.
Performance dimensions Social as well as economic objectives were inherent in early attempts to assist public transit. To these were added the environmental and energy conservation objectives of the 1970s. Therefore, performance analysis for transit must embrace efficiency, effectiveness, as well as equity dimensions. Defining these dimensions, assessing their relative importance and choosing indicators to represent them has been the focus for considerable professional effort. Dajani and Gilbert (1975) outlined a procedure for transportation planners to measure performance from the differing perspectives of federal, state and local officials as well as users and operators. Their conceptual framework distinguished between efficiency, effectiveness and impact measures. Multiple measures are also endorsed by public administrators. Hatry (1978) defines productivity in the public sector as encompassing both efficiency and effectiveness: "Efficiency indicates the extent to which the government produces a given output with the least possible use of resources. Effectiveness indicates the amount of end product, the real service to the public, that the government is providing" (p. 28). Effectiveness includes quality and level of service provided to different groups. Economists distinguish between technical efficiency and productive efficiency and prefer the latter. Technical efficiency is identical with efficiency as used by public administrators: the least units of input required to produce a given level of output. Productive or economic efficiency incorporates the cost of inputs: the maximization of output for a given level of cost or the minimization of the cost of inputs required to produce a given level of output. Improved technical efficiency involves changes in production technology, whereas productive efficiency can be achieved by changes in either prices or amounts of inputs. Both measures are helpful but they should not be confused with cost-effectiveness measures which are frequently used in policy research. Cost-effectiveness measures, like cost per passenger, mix input cost with the consumption of output. They do not measure the cost of output. They provide useful overall measures, but they are difficult to interpret because input cost and consumption of output may not be related. Talley (1988) has argued for cost-effectiveness measures like "maximizing passengers subject to an overall deficit constraint" as the operating objective for public transit firms. Others have suggested single measures like "maximization of passenger miles subject to a budgetary constraint" (Nash, 1978) or "route deficit per passenger" (Talley and Becker, 1982). Such measures may be appropriate for transit systems that normally cover operating cost from passenger fares, but they are
not suitable as separate measures of efficiency and effectiveness for systems that are heavily dependent on governmental assistance. Achieving technical efficiency is much more important.
Performance indicators Consensus regarding the best individual indicator or set of indicators to measure transit performance is unlikely, given the different institutional contexts under which transit operates. A list of 22 indicators appropriate for the efficiency, effectiveness and costeffectiveness dimensions was presented by Fielding, Glauthier, and Lave (1978). In a subsequent study, factor analysis was used with data from the National Urban Mass Transportation Statistics for fiscal years 1981 and 1982 to select a smaller set of marker variables representing each performance dimension. The seven marker variables were: 1. revenue vehicle hours per dollar of operating expense; 2. vehicle miles per peak vehicle; 3. vehicle hours per employee; 4. vehicle miles per maintenance employee; 5. vehicle miles per accident; 6. unlinked passenger trips per revenue vehicle hour; and 7. operating revenue per operating expense (Fielding, Babitsky and Brenner, 1985). The first five represent the efficiency dimension, whereas the last two represent the effectiveness and cost-effectiveness dimensions, respectively. Efficiency was emphasized because of the perceived need to improve this aspect of management in the U.S. Dissatisfaction with the performance of public transit became widespread in the United States in the late 1970s and early 1980s, and performance monitoring was introduced as a solution. Academic research had a profound influence on public policy when legislators sought methodology and indicators to monitor industry progress. Michigan, Pennsylvania, New York, Massachusetts and Florida have each experimented with performance monitoring systems. And each has utilized variations of the aforementioned indicators. Table 1 illustrates how these dimensions are measured in the federal, state and regional evaluation programs considered as case studies in a subsequent section. All three dimensions are represented in each case study. Efficiency is emphasized in the federal triennial audits, whereas effectiveness measures receive more attention in California and Los Angeles. There is general similarity between the overall measures for each dimension. Slight variations occur depending on whether service is measured as total hours of service or total hours of revenue service. Also, the federal reports use revenue vehicle hours (or proportion thereof) per dollar of operating cost. By using the inverse of the normal cost per vehicle service hour,
Transit performance evaluation in the U.S.A.
485
Table 1. Performance indicators used in case studies by performance dimension Dimension Efficiency Overall Labor
Maintenance Effectiveness Overall Safety Cost effectiveness Overall ratio
Federal
California
Revenue vehicle hours per dollar operating cost Vehicle hours per employee Vehicle miles per maintenance employee Vehicle miles per dollar maintenance expense
Operating cost per vehicle service hour Vehicle service hours per employee
Operating cost per vehicle service hour
Passengers per revenue vehicle hour Vehicle miles per collision accident
Passengers per vehicle service hour
Passengers per vehicle service hour
Operating revenue per operating cost
Passenger
Operating cost per passenger
higher numbers indicate improving performance. This aids presentation and interpretation when several indicators are presented on a single page or diagram.
Information system Performance analysis in the U.S. has benefitted from a marvelous data source. In response to Section 15 of the Urban Mass Transportation Act (as amended in 1974), the federal government requires that every agency receiving operating assistance shall report information using the Urban Mass Transpor-
tation Industry Uniform System of Accounts and Records and Reporting System. Comprehensive financial information is submitted according to functional (transportation, maintenance, administration) and object (labor, fringe benefits, fuel, material) categories. Operating data like hours and miles of service, modes operated and employee equivalents are also submitted. Data is compiled, validated and then published annually as the National Mass Urban Transportation Statistics. Since 1983, annual reports have also been made available on diskette in Lotus format. Almost 500 agencies submit information. To encourage use in performance analysis, 29 indicators are calculated separately by mode for agencies with more than 25 vehicles in peak service. Four of the seven aforementioned marker indicators are included; the other three are easily calculated by combining totals from the tables.
Rewarding superior performance There has been little success in using performance measures as incentives for managers and workers. Both Pennsylvania and Michigan have made attempts and then abandoned incentives. Forty-seven indicators were calculated for Michigan operators (Holec, Schwager, and Fandalian, 1980). The mean,
Los Angeles
Operating revenue per operating cost Subsidy per passenger
range and standard deviation for each indicator was calculated and the relative status of each agency publicized. So many indicators were used that the results created confusion. Operators also criticized the use of miles rather than hours as the denominator because miles distorted the performance of agencies operating slower service in central cities. The Michigan program was too ambitious and failed to encourage agencies to improve performance. Pennsylvania has done more than other states to provide incentives (Miller, 1980). Four indicators were calculated for each agency to cover the efficiency and effectiveness dimensions: cost per hour, revenue per hour, ridership per hour, and the revenue-to-expense ratio. The state offered between 66.66 and 75°70 of the assistance required by each agency; the difference, 8.33070, was awarded as a performance incentive if the agency maintained or improved performance over the previous year on the first three measures. To qualify for the incentive, an agency had to exceed a statewide revenue-to-expense ratio. This varied annually between 40 and 45 070. Opposition arose because it was far easier for smaller agencies to qualify for the incentive reward. Elected representatives from Philadelphia and Pittsburgh complained about the loss of assistance to small and midsized cities that were less dependent on transit, and the incentive program was abandoned after seven years. Similar fates befell regional performance incentive schemes proposed for St. Louis and the Delaware Valley. The schemes used too many indicators and operators were able to dispute results and create uncertainty that elected officials were unwilling to arbitrate. A similar outcome will be described in more detail for Los Angeles. Incentive schemes that reward superior performance have not been successful in public transit.
G. J. FIELDING
486
This is a liability. Private contractors are able to reward their managers for achieving performance objectives, and this is one reason why small, private contractors have been price-competitive with large regional operators. They are far less effective when competing with smaller, public operators where the rewards for superior performance are apparent and self-satisfying.
Performance evaluation in perspective Progress was made in developing performance monitoring systems for American transit during the 1980s. The efficiency and effectiveness dimensions created a robust structure, that, when equity considerations were added, provided the basis for measuring progress towards the social and economic goals outlined in legislation. Debate continues over appropriate indicators: operating managers prefer effectiveness indicators that emphasize utilization; economists prefer cost-effectiveness measures that link input cost with consumption; and public administrators favor efficiency indicators that highlight the amount and costs of transit inputs. Differences are appropriate because evaluation strategies serve different purposes. One point is crucial however; the number of indicators should be small and representative of the goals that transit is expected to achieve. Large indicator sets have created disputes over measurement and interpretation. As the case studies in the following section will indicate, even small indicator sets can create problems when the objectives for performance analysis are not explicit. CASE STUDIES
Three evaluation programs are examined. Each represents a different level of government: federal, state and local, with the latter two representative of many different state and local programs implemented during the 1980s. Although the state program in California was initiated prior to the federal program, the latter failed to learn from the California experience. This is unfortunate, because the state program is superior in many respects. The local program in Los Angeles County, CA, went too far; an ambitious monitoring system was introduced with rewards for superior performance. However, it failed to gain support from the operators, was weakened and eventually abandoned.
Federal triennial reviews Recipients of federal transit assistance are required to report their performance annually and certify compliance with federal regulations. Also, since 1982, the Secretary of Transportation has been required to audit, at least every three years, the performance of each recipient in carrying out federal requirements. A precedent had been set as early as 1974; each recipient of federal operating assistance had been required to report financial and operating performance
using a uniform system of accounts and records (U.S. Congress, 1988; UMTA, Section 15). The initial report for the 1978-79 fiscal year appeared in 1982 with subsequent reports appearing annually, approximately 18 months following the close of the fiscal year. To ensure that agencies were complying with the planning and technical requirements of the Urban Mass Transportation Act of 1964 (as amended), Congress required in 1982 that: "the Secretary shall, not less than once every three years, perform a full review and evaluation of the performance of a recipient in carrying out the recipient's program, with specific reference to the compliance with statutory and administrative requirements, and consistency of actual program activities with the proposed program of p r o j e c t s . . . " [U.S. Congress, 1988; UMTA, Section 9(g)(2)]. The Secretary can reduce or withdraw assistance if a recipient is not in compliance. Although Congress was responding to increasing dissatisfaction over the escalation of transit costs, a compliance review rather than an audit of performance was required. This was unfortunate; data on performance was readily available from the Section 15 reports and could have been used to provide an audit of financial and operating performance. Performance data is used in preparation for site visits but not included in the official report (U.S. DOT, 1985). Prior to reviewing an agency, UMTA regional staff use the Section 15 data to calculate seven indicators and prepare time-series profiles for each agency. Performance is compared against that of similar agencies to detect significant changes in operations or maintenance that might indicate problems not anticipated when assistance was awarded. This allows staff to evaluate the effectiveness of federal planning requirements under different operating environments. However, this performance information is neither publicized nor is the transit agency required to explain changes. Privatization of the triennial reviews has reduced and, in several regions, eliminated the analysis of agency performance. Sixty of the 143 reviews conducted in 1991 were performed by private contractors rather than UMTA staff. Compliance with federal rules and regulations is emphasized as a postaudit of compliance, and little attention is given to whether an agency is using federal assistance efficiently and effectively. Occasionally the Section 15 reports are used when the auditor detects a deviation from prevailing maintenance schedules, vehicle inventory, or reporting of passenger mile and vehicle mile statistics. The latter statistics are used to allocate federal operating assistance. Congress required triennial reviews because of concerns about deteriorating performance. But the language of the 1982 amendment called for a review of agency compliance rather than an examination of how federal assistance is used to improve performance. Disagreement over appropriate goals did not allow Congress to pass legislation directing UMTA
Transit performance evaluation in the U.S.A. to audit performance, publicize results, or indicate how performance might be improved. A review of compliance was all that could be achieved.
California performance audits The Californian transit auditing requirement was a response to concerns similar to those that motivated the federal legislation. The procedures, however, were more clearly defined. The goal was to ensure that operating assistance improved the performance of public transit rather than be used by local agencies to decrease fares and expand service into areas with insufficient demand. Unlike the federal reviews, the California legislation requires the tracking of performance and the monitoring of recommended changes: performance indicators were defined in the legislation; deficiencies must be critically examined; remedial programs suggested for implementation by the operating agency; and implementation monitored by a performance committee representing management, employees and users. Operating assistance for transit in California predated the federal program by three years; concern over its effectiveness also came earlier. The Transportation Development Act (TDA) of 1971 extended the sales tax to gasoline, and required local authorities who desired to benefit from the additional revenue to place one-quarter of one cent from the six cent sales tax in a Local Transportation Fund. In counties with more than 500,000 population the fund was exclusively reserved for expenditure on transit and cycleways. In counties with less than 500,000, the fund could be used for streets and highways providing there were no unmet transit needs. A local matching requirement was one of the constraints imposed on applicants for TDA funds. For operating assistance, a local match of 50% was required. Proponents of this requirement argued that it served several purposes: it would prevent the displacement of local subsidies and fare revenue by state subsidy; it would maintain the interest of local elected officials in the prudent management of their system; it would save money that could be used to match federal grants for capital investments; and it would limit the proliferation of inefficient routes to satisfy local political constituencies (Lamare, 1981). Many counties, however, developed loopholes in the legislation to avoid paying their local share. Routes proliferated and the 25 cent systemwide fare became the norm. State officials who had supported the TDA legislation were alarmed by declining performance and sought to mandate attention to performance. The TDA was amended in 1978. The local share of cost requirement was deleted and replaced by stringent reporting requirements (California Public Utility Code, Section 99246 c). The legislation directed that independent auditors must review performance every three years in two phases. Phase I reviews performance since the previous audit and verifies for each operator:
1. 2. 3. 4. 5.
487
operating cost per passenger; operating cost per vehicle service hour; passengers per vehicle service hour; passengers per vehicle service mile; and vehicle service hours per employee.
Phase II includes a more detailed analysis of any functional area where deteriorating performance is observed (Caltrans, 1982). Among the functional areas that are most frequently analyzed are: operations (absenteeism and safety); maintenance staffing; and administrative cost. Audit reports are presented to the directors of the operating agency, to the regional or county agency administering the Local Transportation Fund, and to a local Transit Performance Review Committee. The latter is a voluntary organization that includes transit officials, users of transit, and representatives from affected labor unions. Their function is to review the audit and determine ways in which performance can be improved. Failure to comply with any section of the state regulations may disqualify an agency from receiving state transit assistance. The California transit audit program was designed to achieve improved performance with the minimum of interference from state officials. It reveals to elected officials the accomplishments of their transit agencies and relies upon self-interest and the diligence of the transit performance review committee to achieve improvement.
Los Angeles performance incentive program Performance analysis for transit operators in Los Angeles County is similar to the TDA audits. Several of the same indicators must be reported with the addition of incentive payments to encourage improved performance. The outcome has been disappointing. Whereas the decision to insert private-market incentives into the public-subsidy domain was regarded as innovative at the beginning of the 1980s, the program was virtually abandoned by the end of the decade. Calculating the performance incentives required too much interference with management, and operators resisted the program at every opportunity. Competition among local governments undercut support, and elected officials disliked the program: it disturbed existing arrangements and compelled them to penalize part of their own constituency. A complex political arrangement shrouds transit in Los Angeles County. Fourteen operators provide public transit service: three operate fixed-route, fixed-schedule bus service; six operate both fixedroute and demand-response service; while five operate only demand-response service. Only the Southern California Rapid Transit District (RTD), the largest agency that operates 86% of all service, is an independent agency. The remainder are operated by municipalities. Allocation of federal and state assistance, as well as local sales taxes devoted to transit, is controlled by the Los Angeles County Transportation Commission (LACTC). The Commission also
488
G.J. FIELDING
complies the short-range transportation plan, but responsibility for long-range planning belongs to the Southern California Association of Governments (SCAG). To add to the complexity, the State Legislature had mandated that rail rapid transit be jointly planned by RTD and L A C T C and operated by RTD. And in 1991 the governing boards were required to have overlapping membership: six members from the RTD Board now have seats on the eleven member Commission (California Public Utility Code, Section 130051.5). Increasing operating costs for transit during the 1970s, and the unwillingness of operators to coordinate services, prompted the State Legislature to mandate change in 1979. The performance audits, required for all operators in California, provided insufficient incentive for change, so the Legislature chose to insert private-market incentives. Assembly Bill 103 directed LACTC to allocate state and federal funding among the operators so as to encourage improved performance. The objective was to shift service from the less efficient to the more efficient providers. To achieve this, the Commission was also required to establish a "transportation coordination and service program" that was to "promote the efficient and effective use of all available transportation resources in the c o u n t y . . . " The coordination program was to include, among other things: transit productivity guidelines and specific steps to be taken to bring existing transit services into conformity with the guidelines and financial standards which all transit operators in the county were to meet (California Public Utility Code, Section 130380). The Commission was also given the power to reallocate state assistance to different areas and new operators. These requirements provided the justification for what came to be called the Transportation Performance Measurement (TPM) Program. The legislation required the Commission to develop and adopt their performance program by 1981 and to revise it biennially thereafter. The revision requirement is important because it led to changes that reduced and eventually abolished incentives. The initial TPM required operators to earn part of their operating assistance by exceeding performance standards: 95% of the federal, state and local assistance was to be allocated by formula leaving 5% ($12 million in 1982) as an incentive pool. Service was allocated into six categories to distinguish among the different types of service provided. Performance measures were calculated for each category and eligibility for incentive payments determined by meeting or exceeding four standards: 1. operating cost per vehicle service hour: increase from year to year should not exceed the actual rate of price inflation measured by the consumer price index for Los Angeles. 2. ratio of operating revenue, plus local subsidies, to operating cost: the ratio should not be less than one-third.
3. subsidy per passenger: operating subsidy requested from LACTC per unlinked passenger should not exceed 133°70 of the mean for that category of service for all operators in the county for that year. 4. passengers per hour: the number of passengers per weekday vehicle service hour for local and express service shall not be fewer than 30; for demandresponsive, not fewer than five; and for handicapped service, not fewer than 2.5. Operators were eligible to receive 25% of their proportional bonus by exceeding each standard. After 1985, this was revised to give 40% of the bonus based upon the operating cost standard and 20% each to the remaining three. Shares in the bonus pool were determined by service miles operated. For example, Long Beach operated 4.8% of the vehicle service miles, and this determined their maximum share of the bonus pool (Fielding, Mundle, and Misner, 1982). Calculating performance by service category was complex. The TPM required the calculation of four performance standards for six service categories. Almost a year of meetings were required before operators would agree on the categories. And to add to the complexity, the standards were calculated after the agencies had submitted their audited statements to the Commission. The Commission allocated assistance based upon estimates, but had to wait almost two years before assessing penalties or rewards on money that the operator had received and presumably spent. The TPM Program was intended to improve performance, but the results arrived too late. When the results for 1981-82 were calculated in 1984, operators were alarmed by the magnitude of their penalties (Table 2). The SCRTD was penalized almost $5 million and several of the smaller operators were liable for penalties that exceeded the 5°70 upper bound. Operators feared that savings in the incentive pool would be used by the Commission to encourage competition from private contractors. As release of the data coincided with the biennial reauthorization of the program, operators argued that the program provided no incentive and should be changed. Commissioners agreed, and the TPM program was revised in 1985 in the operators' favor. Henceforth, the incentive pool was to be restricted to a percentage of the local sales tax funds. State and federal operating subsidies would be distributed to operators by formula together with 90% of the anticipated local sales-tax receipts. The remaining 10% could be "earned" each year based upon the operator's historical performance on the four performance measures. In other words, past performance would be rewarded using an incentive dividend. And any unused portion of the incentive pool would remain in the pool to be divided among the operators in subsequent years; it would not be available to assist competing public or private suppliers.
Transit performance evaluation in the U.S.A.
489
Table 2. Penalties calculated under original TPM programs ($) Penalties
Penalties
Based on FY 81-82
Based o n FY 82-83
Upper
Performance Arcadia Claremont Commerce Culver City Gardena H e r m o s a Beach
La Mirada Long Beach Montebello
Norwalk R e d o n d o Beach Santa M o n i c a SCRTD Torrance Total
Bound *
--
1,050 40,382 31,649 21,261 7,526 18,281 148,129 50,979 167,295 ---
4,920,504 26,981 5,434,037
8,300
1,700 22,650 70,700 103,100 2,300 11,050 546,750 121,700 63,550 1,100 434,750
9,255,900 102,050 10,745,600
Performance --
m 4,031 112,472 ---
m -36,046 69,307 ---
13,687,062 79,428 13,988,346
Upper Bound *
8,400 2,100 23,450 69,300 105,750 1,100
6,350 566,250 127,650 56,300 1,000 517,400
10,141,350 106,400 11,732,800
*The u p p e r b o u n d for e a c h o p e r a t o r is 5% o f its state and federal o p e r a t i n g subsidies. Penalties w e r e to be withdrawn from future funding. Source: Los A n g e l e s C o u n t y Transportation Commission, 1985.
Not only were the operators successful in eliminating the threat of competition, but they also preserved any "unearned b o n u s e s " - penalty reduct i o n s - in the revised incentive pool. These changes reduced the incentive to improve performance: first, by eliminating the threat of competition from private companies, and second, by allowing agencies to recapture the penalty in subsequent years. For example, Long Beach Transit, which earned only $416,390 of their possible bonus of $625,680 in 1986-87, or 66.55%0, was able to regain $111,827 by 1987; 84.42% of their upper bound. SCRTD recaptured $2.23 from their $2.77 million penalty for the same period. Penalties still existed, but they were ineffective. The TPM continued until 1990. Although performance was calculated against standards and penalties and bonuses assessed, there was little incentive for the largest operators to improve their management. A small percentage of "their" money was shifted to smaller operators, but this was insufficient to change managerial behavior. In 1990, six members of the SCRTD Board of Directors became Commissioners of LACTC as a result of another reorganization plan initiated by the State Legislature. As SCRTD produces 86% of the bus service, the Commissioners have indicated that they would be awarding incentives to their own agency and wished to eliminate the incentive program. The Commission, however, must continue to monitor performance in accordance with the 1979 legislation. The incentive program for transit in Pennsylvania had a similar fate (Miller, 1980). Incentives for superior performance were introduced in 1979 and later abandoned. The larger operators were penalized for
deteriorating performance and lobbied against the program. Like the operators in Los Angeles, t h o s e in Pennsylvania are now required to report performance without incentive or penalty assessments. The California TDA audits, which are also required from transit operators in Los Angeles County, have been more influential on managerial behavior. Both management and the board of directors are required to consider changes recommended by the auditor. Proposals are also reviewed by the countywide performance review committee, and progress must be reconsidered in subsequent audits. Public discussion of potential deficiencies has had more influence on managerial behavior than potential loss of a small bonus. When there is a close association between the regulatory agency and regulated operators, it is not uncommon for the operators to succeed by capturing the agency. Economists observing this pattern in federal agencies refer to it as regulatory capture (Stigler, 1971). The LACTC staff began the TPM program armed with a consultant's report and reinforced by a mandate from the Legislature. Commissioners were less enthusiastic; they recognized that improved performance would require reduction of service in low density areas and a lower real growth in salaries and benefits for transit employees. As elected officials, they sought to avoid situations that required them to reward one group at the expense of another; they preferred a solution that gave the appearance of rewarding all groups. Transit managers were opposed; they desired the least interference and lobbied both the staff and commissioners to minimize requirements. The LACTC staff were required to meet fre-
G. J. FIELDING
490
quently with the operators in order to design a performance monitoring and incentive program that would satisfy the operators and yet fulfill the legislative requirement. Interaction within the network tended to gravitate towards a consensus; and, because the outcome was more important to the operators than it was to the L A C T C staff, the views of the operators prevailed. The 1981 T P M program was adopted but never enforced. The staff who helped to design the initial program moved on, some to operating agencies, and their replacements were indifferent to the T P M program and more interested in working with the operators. The T P M program was modified in 1985 and the incentive portion dropped in 1990.
CONCLUSION Although performance-based approaches to management have helped agencies focus on cost control during the 1980s, the problems have not been solved. Far too much effort continues to be devoted pursuing markets in which transit cannot be effective. Performance is improving, however, and performance evaluation programs deserve some of the credit. In the largest agencies, real cost-per-revenue vehicle hour for bus service has increased by less than 1 o70 per annum between 1986 and 1989. These large agencies, with more than 250 vehicles in peak-hour service, operate two-thirds of all bus service. Much still remains to be accomplished through diffusion of successful practices. O f the three case studies, only Californian T D A audits can be regarded as a success. These rely on a small set of diagnostic indicators to reveal progress over three years and then use the information to identify causes for deterioration. Agencies have three years to correct adverse reports. In addition, agencies are required to discuss the report, plus any remedial actions, with the performance committee representing groups which will be affected by the changes. Employee representatives serve on this committee. The incentive program designed for Los Angeles County has been ineffective. Too much effort was required to calculate the indices and decide penalties and rewards. A n d the incentives came too late for any beneficial change. It represented a novel attempt to introduce private incentives that overlooked the complexity of public management. I m p r o v e m e n t in transit performance requires cooperation between labor, management and the governing board rather than conflict. The federal triennial reviews do not go far enough. U n i f o r m data are used, but there is no requirement that a small set of indicators be reported to all agencies. Agencies are not required to act upon recommendations unless the agency has failed to comply with federal regulations. The federal review could be more effective if modified to incorporate features of the Californian audits.
Failure of incentive programs has been a disappointment. The experience in Michigan, Pennsylvania and Los Angeles suggests that failure occurs because allowance is not made for differences in operating policies. Large agencies operate regional service seven clays a week and almost 24 hours a day. Service is slower and more expensive to produce than community-based service operated by municipal systems. And when measured against statewide or regional standards, the regional agency appears less efficient, although they may be transporting many more passengers per service hour. They are penalized for assisting those individuals and areas that transit policy was originally designed to help. Incentive programs have been more helpful within agencies. Coach operators are rewarded for reliable attendance and safe driving. And private contractors have been offered cash incentives if they earn more than expected fare revenue or provide service at a lower cost. But cash incentives are not granted to public managers. For example, managers are not rewarded for reducing costs below budget or operating their maintenance departments efficiently. Public administrators are expected to achieve these economies; to reward them financially might be regarded as a gift of public funds. Employees in public agencies would be more likely to make the difficult decisions required to improve efficiency if they shared in the resulting benefits. New indicators and procedures for measurement would be required, as well as different organizational arrangements. Such requirements provide a potential direction for future research in performance evaluation.
REFERENCES
Caltrans (1982) Transit Performance Audit Guidebook. State Department of Transportation, Sacramento, CA. Dajani J. S., Gilbert G. (1978) Measuring the performance of transit systems. Transpn, Plann. Technol., 4, 2, 97 103. Fielding G. J., Glauthier R. E., Lave C. A. (1978) Performance indicators for transit management. Transportation, 7, 4, 365-379. Fielding G. J., Mundle S. R., Misner J. (1982) Performance based funding allocation guidelines for transit operations in Los Angeles county. Transpn. Res. Rec., 857, 14-18. Fielding G. J., Babitsky T. T., Brenner M. E. (1985) Performance analysis for bus transit. Transpn. Res. A, 19A, 1, 73-82. Fielding G. J. (1987) Managing Public Transit Strategically. Jossey-Bass, San Francisco, CA. Hatry H. P. (1978) The status of productivity measurement in the public sector. Public Admin. Rev., 38, 1, 28-33. Holec J. M., Schwager D. S., Fandalian A. (1980) Use of federal Section 15 data in transit performance: Michigan program. Transpn. Res. Rec., 765, 36-38. Lamare J. L. (1981) Intergovernmental finance, productivity, and the local match question: The case of California's transit subsidy policy. Public Admin. Rev., 41, 4, 463-470. Miller J. H. (1980) The use of performance-based methodologies for the allocation of transit operating funds. Traffic Q., 34, 4, 555-585.
Transit performance evaluation in the U.S.A. Nash C. A. 0978) Management objectives in bus transport. J. Transport Econom. Policy, 12, 1, 70-85. Smerk G. M. (1991) The Federal Role in Urban Mass Transportation. Indiana University Press, Bloomington, IN. Stigler G. (1971) The theory of economic regulation. Bell J. Econom., Spring, 3-21. Talley W. K. (1988) An economic theory of the public transit firm. Transpn. Res., 22B, 1, 45-54. Talley W. K., Becker J. (1982) A single measure for evaluating public transit systems. Transpn. Q., 36, 3,423-431.
491
U.S. Congress, Committee on Public Works and Transportation (1988) Compilation of Selected Surface Transportation Laws: Title 23, United States Code. U.S. Government Printing Office, Washington, DC. U.S. DOT (1991) Section 9 Triennial Reviews. UMTA Notice, N9010.1, Washington, DC. U.S. DOT (1991) Public Transportation in the United States: Performance and Condition. The Secretary of Transportation's Report to the U.S. Congress Pursuant to 49USC308.
0191-2607/92 S5.00 + .00 © 1992 Pergamon Press Ltd.
T R A N S I T P E R F O R M A N C E E V A L U A T I O N IN THE U.S.A. GORDON J. FIELDING University of California, School of Social Sciences, Irvine, CA 92717, U.S.A. Abstract-Performance of transit agencies in the United States improved during the 1980s. At the beginning of the decade, Americans had become disenchanted with transit; legislation was passed that required agencies to report performance and accept regular audits. Theory underlying these policies is examined in four components: dimensions for policy objectives, indicators, information systems and incentives. Three programs are examined: federal triennial reviews that monitor compliance with planning and grant requirements, California performance audits that analyze goals and track performance on five indicators and the Los Angeles program that encourages improvement by offering incentive payments for better-than-average performance. The California audits have been the most successful.
INTRODUCTION
Improving public transport is an art; we are not sure what works, because systematic evaluation of performance against intended goals has not been conducted for metropolitan areas. Studies have focused on single modes, overlooked the goal of increasing highway capacity, and emphasized efficiency when efficiency was not mentioned as a goal for transit until the mid 1970s. Even the evaluations mandated by federal, state and local legislation in the United States are deficient: objectives are not adequately defined and measurement criteria do not indicate standards against which performance might be assessed. Evaluation of public transit is made difficult by changing goals. Preserving the commercial advantages of central cities and the jobs of transit employees was initially important; then social, environmental and conservation goals were added in the 1970s (Fielding, 1987). Federal investment in public transit began when older, central cities became alarmed over the rate of federal investment in highways and the abandonment of commuter service by railroads (Smerk, 1991). To this were added a whole host of social and environmental goals during the 1970s that transit alone was incapable of achieving. As management has struggled to respond to these changing goals, performance of transit has declined. Public transit in the U.S., which had been covering operating costs from the fare box until 1965, was recovering only 54070 in 1975 and 42070 in 1985. For the fiscal year ending in 1989, the annual operating deficit was $7.1 billion, 48070 of costs. And transit ridership was virtually unchanged despite the generosity of federal, state and local governments. Transit had reversed the downward trend of ridership and cost recovery, and had maintained its share of travel to work, but all this had been achieved at a very high cost per passenger. Alarm over operating deficits caused legislators to seek different solutions. Performance reviews were mandated in the early 1980s as a way to force
some discipline into public transit while continuing the earlier expectations. As a result, cost increases have been controlled and operating revenue increased: the ratio of operating revenue to cost has increased from 38°7o in 1980 to 42070 in 1985, and 48070 in 1989; real cost increases in the last half of the 1980s were less than 1070per annum compared to increases of between 4 and 507o over inflation between 1970 and 1985 (U.S. DOT, 1991). Performance auditing cannot claim exclusive credit for cost control, but it has focused attention on the unbridled expenditures of the 1970s and the need for improved performance in every agency. Three performance programs will be reviewed: the federal requirement for triennial audits, the California performance audits for operators receiving assistance from the Transportation Development Act and the performance incentive program utilized by the Los Angeles Country Transportation Commission. Each will be judged in terms of intentions, techniques of measurement, and use of results to improve performance. PERFORMANCE THEORY
Performance evaluation of public enterprise is based on the premise that administration needs to be untangled; that a small set of indicators can be used to distinguish "good" from "bad" performance. Indicators provide a "bottom line" that, although not as generally accepted as those used by private enterprise, nevertheless provide measures helpful when gauging success or failure. Systems for performance evaluation require four components: 1. dimensions that represent the objectives that motivated public intervention; 2. indicators that translate objectives into quantitative measures; 3. an information system that gathers appropriate data in a consistent manner to provide crosssectional and time-series statistics; and
483
484
G, J. FIELDING
4. an incentive system that rewards managers for improving performance.
Performance dimensions Social as well as economic objectives were inherent in early attempts to assist public transit. To these were added the environmental and energy conservation objectives of the 1970s. Therefore, performance analysis for transit must embrace efficiency, effectiveness, as well as equity dimensions. Defining these dimensions, assessing their relative importance and choosing indicators to represent them has been the focus for considerable professional effort. Dajani and Gilbert (1975) outlined a procedure for transportation planners to measure performance from the differing perspectives of federal, state and local officials as well as users and operators. Their conceptual framework distinguished between efficiency, effectiveness and impact measures. Multiple measures are also endorsed by public administrators. Hatry (1978) defines productivity in the public sector as encompassing both efficiency and effectiveness: "Efficiency indicates the extent to which the government produces a given output with the least possible use of resources. Effectiveness indicates the amount of end product, the real service to the public, that the government is providing" (p. 28). Effectiveness includes quality and level of service provided to different groups. Economists distinguish between technical efficiency and productive efficiency and prefer the latter. Technical efficiency is identical with efficiency as used by public administrators: the least units of input required to produce a given level of output. Productive or economic efficiency incorporates the cost of inputs: the maximization of output for a given level of cost or the minimization of the cost of inputs required to produce a given level of output. Improved technical efficiency involves changes in production technology, whereas productive efficiency can be achieved by changes in either prices or amounts of inputs. Both measures are helpful but they should not be confused with cost-effectiveness measures which are frequently used in policy research. Cost-effectiveness measures, like cost per passenger, mix input cost with the consumption of output. They do not measure the cost of output. They provide useful overall measures, but they are difficult to interpret because input cost and consumption of output may not be related. Talley (1988) has argued for cost-effectiveness measures like "maximizing passengers subject to an overall deficit constraint" as the operating objective for public transit firms. Others have suggested single measures like "maximization of passenger miles subject to a budgetary constraint" (Nash, 1978) or "route deficit per passenger" (Talley and Becker, 1982). Such measures may be appropriate for transit systems that normally cover operating cost from passenger fares, but they are
not suitable as separate measures of efficiency and effectiveness for systems that are heavily dependent on governmental assistance. Achieving technical efficiency is much more important.
Performance indicators Consensus regarding the best individual indicator or set of indicators to measure transit performance is unlikely, given the different institutional contexts under which transit operates. A list of 22 indicators appropriate for the efficiency, effectiveness and costeffectiveness dimensions was presented by Fielding, Glauthier, and Lave (1978). In a subsequent study, factor analysis was used with data from the National Urban Mass Transportation Statistics for fiscal years 1981 and 1982 to select a smaller set of marker variables representing each performance dimension. The seven marker variables were: 1. revenue vehicle hours per dollar of operating expense; 2. vehicle miles per peak vehicle; 3. vehicle hours per employee; 4. vehicle miles per maintenance employee; 5. vehicle miles per accident; 6. unlinked passenger trips per revenue vehicle hour; and 7. operating revenue per operating expense (Fielding, Babitsky and Brenner, 1985). The first five represent the efficiency dimension, whereas the last two represent the effectiveness and cost-effectiveness dimensions, respectively. Efficiency was emphasized because of the perceived need to improve this aspect of management in the U.S. Dissatisfaction with the performance of public transit became widespread in the United States in the late 1970s and early 1980s, and performance monitoring was introduced as a solution. Academic research had a profound influence on public policy when legislators sought methodology and indicators to monitor industry progress. Michigan, Pennsylvania, New York, Massachusetts and Florida have each experimented with performance monitoring systems. And each has utilized variations of the aforementioned indicators. Table 1 illustrates how these dimensions are measured in the federal, state and regional evaluation programs considered as case studies in a subsequent section. All three dimensions are represented in each case study. Efficiency is emphasized in the federal triennial audits, whereas effectiveness measures receive more attention in California and Los Angeles. There is general similarity between the overall measures for each dimension. Slight variations occur depending on whether service is measured as total hours of service or total hours of revenue service. Also, the federal reports use revenue vehicle hours (or proportion thereof) per dollar of operating cost. By using the inverse of the normal cost per vehicle service hour,
Transit performance evaluation in the U.S.A.
485
Table 1. Performance indicators used in case studies by performance dimension Dimension Efficiency Overall Labor
Maintenance Effectiveness Overall Safety Cost effectiveness Overall ratio
Federal
California
Revenue vehicle hours per dollar operating cost Vehicle hours per employee Vehicle miles per maintenance employee Vehicle miles per dollar maintenance expense
Operating cost per vehicle service hour Vehicle service hours per employee
Operating cost per vehicle service hour
Passengers per revenue vehicle hour Vehicle miles per collision accident
Passengers per vehicle service hour
Passengers per vehicle service hour
Operating revenue per operating cost
Passenger
Operating cost per passenger
higher numbers indicate improving performance. This aids presentation and interpretation when several indicators are presented on a single page or diagram.
Information system Performance analysis in the U.S. has benefitted from a marvelous data source. In response to Section 15 of the Urban Mass Transportation Act (as amended in 1974), the federal government requires that every agency receiving operating assistance shall report information using the Urban Mass Transpor-
tation Industry Uniform System of Accounts and Records and Reporting System. Comprehensive financial information is submitted according to functional (transportation, maintenance, administration) and object (labor, fringe benefits, fuel, material) categories. Operating data like hours and miles of service, modes operated and employee equivalents are also submitted. Data is compiled, validated and then published annually as the National Mass Urban Transportation Statistics. Since 1983, annual reports have also been made available on diskette in Lotus format. Almost 500 agencies submit information. To encourage use in performance analysis, 29 indicators are calculated separately by mode for agencies with more than 25 vehicles in peak service. Four of the seven aforementioned marker indicators are included; the other three are easily calculated by combining totals from the tables.
Rewarding superior performance There has been little success in using performance measures as incentives for managers and workers. Both Pennsylvania and Michigan have made attempts and then abandoned incentives. Forty-seven indicators were calculated for Michigan operators (Holec, Schwager, and Fandalian, 1980). The mean,
Los Angeles
Operating revenue per operating cost Subsidy per passenger
range and standard deviation for each indicator was calculated and the relative status of each agency publicized. So many indicators were used that the results created confusion. Operators also criticized the use of miles rather than hours as the denominator because miles distorted the performance of agencies operating slower service in central cities. The Michigan program was too ambitious and failed to encourage agencies to improve performance. Pennsylvania has done more than other states to provide incentives (Miller, 1980). Four indicators were calculated for each agency to cover the efficiency and effectiveness dimensions: cost per hour, revenue per hour, ridership per hour, and the revenue-to-expense ratio. The state offered between 66.66 and 75°70 of the assistance required by each agency; the difference, 8.33070, was awarded as a performance incentive if the agency maintained or improved performance over the previous year on the first three measures. To qualify for the incentive, an agency had to exceed a statewide revenue-to-expense ratio. This varied annually between 40 and 45 070. Opposition arose because it was far easier for smaller agencies to qualify for the incentive reward. Elected representatives from Philadelphia and Pittsburgh complained about the loss of assistance to small and midsized cities that were less dependent on transit, and the incentive program was abandoned after seven years. Similar fates befell regional performance incentive schemes proposed for St. Louis and the Delaware Valley. The schemes used too many indicators and operators were able to dispute results and create uncertainty that elected officials were unwilling to arbitrate. A similar outcome will be described in more detail for Los Angeles. Incentive schemes that reward superior performance have not been successful in public transit.
G. J. FIELDING
486
This is a liability. Private contractors are able to reward their managers for achieving performance objectives, and this is one reason why small, private contractors have been price-competitive with large regional operators. They are far less effective when competing with smaller, public operators where the rewards for superior performance are apparent and self-satisfying.
Performance evaluation in perspective Progress was made in developing performance monitoring systems for American transit during the 1980s. The efficiency and effectiveness dimensions created a robust structure, that, when equity considerations were added, provided the basis for measuring progress towards the social and economic goals outlined in legislation. Debate continues over appropriate indicators: operating managers prefer effectiveness indicators that emphasize utilization; economists prefer cost-effectiveness measures that link input cost with consumption; and public administrators favor efficiency indicators that highlight the amount and costs of transit inputs. Differences are appropriate because evaluation strategies serve different purposes. One point is crucial however; the number of indicators should be small and representative of the goals that transit is expected to achieve. Large indicator sets have created disputes over measurement and interpretation. As the case studies in the following section will indicate, even small indicator sets can create problems when the objectives for performance analysis are not explicit. CASE STUDIES
Three evaluation programs are examined. Each represents a different level of government: federal, state and local, with the latter two representative of many different state and local programs implemented during the 1980s. Although the state program in California was initiated prior to the federal program, the latter failed to learn from the California experience. This is unfortunate, because the state program is superior in many respects. The local program in Los Angeles County, CA, went too far; an ambitious monitoring system was introduced with rewards for superior performance. However, it failed to gain support from the operators, was weakened and eventually abandoned.
Federal triennial reviews Recipients of federal transit assistance are required to report their performance annually and certify compliance with federal regulations. Also, since 1982, the Secretary of Transportation has been required to audit, at least every three years, the performance of each recipient in carrying out federal requirements. A precedent had been set as early as 1974; each recipient of federal operating assistance had been required to report financial and operating performance
using a uniform system of accounts and records (U.S. Congress, 1988; UMTA, Section 15). The initial report for the 1978-79 fiscal year appeared in 1982 with subsequent reports appearing annually, approximately 18 months following the close of the fiscal year. To ensure that agencies were complying with the planning and technical requirements of the Urban Mass Transportation Act of 1964 (as amended), Congress required in 1982 that: "the Secretary shall, not less than once every three years, perform a full review and evaluation of the performance of a recipient in carrying out the recipient's program, with specific reference to the compliance with statutory and administrative requirements, and consistency of actual program activities with the proposed program of p r o j e c t s . . . " [U.S. Congress, 1988; UMTA, Section 9(g)(2)]. The Secretary can reduce or withdraw assistance if a recipient is not in compliance. Although Congress was responding to increasing dissatisfaction over the escalation of transit costs, a compliance review rather than an audit of performance was required. This was unfortunate; data on performance was readily available from the Section 15 reports and could have been used to provide an audit of financial and operating performance. Performance data is used in preparation for site visits but not included in the official report (U.S. DOT, 1985). Prior to reviewing an agency, UMTA regional staff use the Section 15 data to calculate seven indicators and prepare time-series profiles for each agency. Performance is compared against that of similar agencies to detect significant changes in operations or maintenance that might indicate problems not anticipated when assistance was awarded. This allows staff to evaluate the effectiveness of federal planning requirements under different operating environments. However, this performance information is neither publicized nor is the transit agency required to explain changes. Privatization of the triennial reviews has reduced and, in several regions, eliminated the analysis of agency performance. Sixty of the 143 reviews conducted in 1991 were performed by private contractors rather than UMTA staff. Compliance with federal rules and regulations is emphasized as a postaudit of compliance, and little attention is given to whether an agency is using federal assistance efficiently and effectively. Occasionally the Section 15 reports are used when the auditor detects a deviation from prevailing maintenance schedules, vehicle inventory, or reporting of passenger mile and vehicle mile statistics. The latter statistics are used to allocate federal operating assistance. Congress required triennial reviews because of concerns about deteriorating performance. But the language of the 1982 amendment called for a review of agency compliance rather than an examination of how federal assistance is used to improve performance. Disagreement over appropriate goals did not allow Congress to pass legislation directing UMTA
Transit performance evaluation in the U.S.A. to audit performance, publicize results, or indicate how performance might be improved. A review of compliance was all that could be achieved.
California performance audits The Californian transit auditing requirement was a response to concerns similar to those that motivated the federal legislation. The procedures, however, were more clearly defined. The goal was to ensure that operating assistance improved the performance of public transit rather than be used by local agencies to decrease fares and expand service into areas with insufficient demand. Unlike the federal reviews, the California legislation requires the tracking of performance and the monitoring of recommended changes: performance indicators were defined in the legislation; deficiencies must be critically examined; remedial programs suggested for implementation by the operating agency; and implementation monitored by a performance committee representing management, employees and users. Operating assistance for transit in California predated the federal program by three years; concern over its effectiveness also came earlier. The Transportation Development Act (TDA) of 1971 extended the sales tax to gasoline, and required local authorities who desired to benefit from the additional revenue to place one-quarter of one cent from the six cent sales tax in a Local Transportation Fund. In counties with more than 500,000 population the fund was exclusively reserved for expenditure on transit and cycleways. In counties with less than 500,000, the fund could be used for streets and highways providing there were no unmet transit needs. A local matching requirement was one of the constraints imposed on applicants for TDA funds. For operating assistance, a local match of 50% was required. Proponents of this requirement argued that it served several purposes: it would prevent the displacement of local subsidies and fare revenue by state subsidy; it would maintain the interest of local elected officials in the prudent management of their system; it would save money that could be used to match federal grants for capital investments; and it would limit the proliferation of inefficient routes to satisfy local political constituencies (Lamare, 1981). Many counties, however, developed loopholes in the legislation to avoid paying their local share. Routes proliferated and the 25 cent systemwide fare became the norm. State officials who had supported the TDA legislation were alarmed by declining performance and sought to mandate attention to performance. The TDA was amended in 1978. The local share of cost requirement was deleted and replaced by stringent reporting requirements (California Public Utility Code, Section 99246 c). The legislation directed that independent auditors must review performance every three years in two phases. Phase I reviews performance since the previous audit and verifies for each operator:
1. 2. 3. 4. 5.
487
operating cost per passenger; operating cost per vehicle service hour; passengers per vehicle service hour; passengers per vehicle service mile; and vehicle service hours per employee.
Phase II includes a more detailed analysis of any functional area where deteriorating performance is observed (Caltrans, 1982). Among the functional areas that are most frequently analyzed are: operations (absenteeism and safety); maintenance staffing; and administrative cost. Audit reports are presented to the directors of the operating agency, to the regional or county agency administering the Local Transportation Fund, and to a local Transit Performance Review Committee. The latter is a voluntary organization that includes transit officials, users of transit, and representatives from affected labor unions. Their function is to review the audit and determine ways in which performance can be improved. Failure to comply with any section of the state regulations may disqualify an agency from receiving state transit assistance. The California transit audit program was designed to achieve improved performance with the minimum of interference from state officials. It reveals to elected officials the accomplishments of their transit agencies and relies upon self-interest and the diligence of the transit performance review committee to achieve improvement.
Los Angeles performance incentive program Performance analysis for transit operators in Los Angeles County is similar to the TDA audits. Several of the same indicators must be reported with the addition of incentive payments to encourage improved performance. The outcome has been disappointing. Whereas the decision to insert private-market incentives into the public-subsidy domain was regarded as innovative at the beginning of the 1980s, the program was virtually abandoned by the end of the decade. Calculating the performance incentives required too much interference with management, and operators resisted the program at every opportunity. Competition among local governments undercut support, and elected officials disliked the program: it disturbed existing arrangements and compelled them to penalize part of their own constituency. A complex political arrangement shrouds transit in Los Angeles County. Fourteen operators provide public transit service: three operate fixed-route, fixed-schedule bus service; six operate both fixedroute and demand-response service; while five operate only demand-response service. Only the Southern California Rapid Transit District (RTD), the largest agency that operates 86% of all service, is an independent agency. The remainder are operated by municipalities. Allocation of federal and state assistance, as well as local sales taxes devoted to transit, is controlled by the Los Angeles County Transportation Commission (LACTC). The Commission also
488
G.J. FIELDING
complies the short-range transportation plan, but responsibility for long-range planning belongs to the Southern California Association of Governments (SCAG). To add to the complexity, the State Legislature had mandated that rail rapid transit be jointly planned by RTD and L A C T C and operated by RTD. And in 1991 the governing boards were required to have overlapping membership: six members from the RTD Board now have seats on the eleven member Commission (California Public Utility Code, Section 130051.5). Increasing operating costs for transit during the 1970s, and the unwillingness of operators to coordinate services, prompted the State Legislature to mandate change in 1979. The performance audits, required for all operators in California, provided insufficient incentive for change, so the Legislature chose to insert private-market incentives. Assembly Bill 103 directed LACTC to allocate state and federal funding among the operators so as to encourage improved performance. The objective was to shift service from the less efficient to the more efficient providers. To achieve this, the Commission was also required to establish a "transportation coordination and service program" that was to "promote the efficient and effective use of all available transportation resources in the c o u n t y . . . " The coordination program was to include, among other things: transit productivity guidelines and specific steps to be taken to bring existing transit services into conformity with the guidelines and financial standards which all transit operators in the county were to meet (California Public Utility Code, Section 130380). The Commission was also given the power to reallocate state assistance to different areas and new operators. These requirements provided the justification for what came to be called the Transportation Performance Measurement (TPM) Program. The legislation required the Commission to develop and adopt their performance program by 1981 and to revise it biennially thereafter. The revision requirement is important because it led to changes that reduced and eventually abolished incentives. The initial TPM required operators to earn part of their operating assistance by exceeding performance standards: 95% of the federal, state and local assistance was to be allocated by formula leaving 5% ($12 million in 1982) as an incentive pool. Service was allocated into six categories to distinguish among the different types of service provided. Performance measures were calculated for each category and eligibility for incentive payments determined by meeting or exceeding four standards: 1. operating cost per vehicle service hour: increase from year to year should not exceed the actual rate of price inflation measured by the consumer price index for Los Angeles. 2. ratio of operating revenue, plus local subsidies, to operating cost: the ratio should not be less than one-third.
3. subsidy per passenger: operating subsidy requested from LACTC per unlinked passenger should not exceed 133°70 of the mean for that category of service for all operators in the county for that year. 4. passengers per hour: the number of passengers per weekday vehicle service hour for local and express service shall not be fewer than 30; for demandresponsive, not fewer than five; and for handicapped service, not fewer than 2.5. Operators were eligible to receive 25% of their proportional bonus by exceeding each standard. After 1985, this was revised to give 40% of the bonus based upon the operating cost standard and 20% each to the remaining three. Shares in the bonus pool were determined by service miles operated. For example, Long Beach operated 4.8% of the vehicle service miles, and this determined their maximum share of the bonus pool (Fielding, Mundle, and Misner, 1982). Calculating performance by service category was complex. The TPM required the calculation of four performance standards for six service categories. Almost a year of meetings were required before operators would agree on the categories. And to add to the complexity, the standards were calculated after the agencies had submitted their audited statements to the Commission. The Commission allocated assistance based upon estimates, but had to wait almost two years before assessing penalties or rewards on money that the operator had received and presumably spent. The TPM Program was intended to improve performance, but the results arrived too late. When the results for 1981-82 were calculated in 1984, operators were alarmed by the magnitude of their penalties (Table 2). The SCRTD was penalized almost $5 million and several of the smaller operators were liable for penalties that exceeded the 5°70 upper bound. Operators feared that savings in the incentive pool would be used by the Commission to encourage competition from private contractors. As release of the data coincided with the biennial reauthorization of the program, operators argued that the program provided no incentive and should be changed. Commissioners agreed, and the TPM program was revised in 1985 in the operators' favor. Henceforth, the incentive pool was to be restricted to a percentage of the local sales tax funds. State and federal operating subsidies would be distributed to operators by formula together with 90% of the anticipated local sales-tax receipts. The remaining 10% could be "earned" each year based upon the operator's historical performance on the four performance measures. In other words, past performance would be rewarded using an incentive dividend. And any unused portion of the incentive pool would remain in the pool to be divided among the operators in subsequent years; it would not be available to assist competing public or private suppliers.
Transit performance evaluation in the U.S.A.
489
Table 2. Penalties calculated under original TPM programs ($) Penalties
Penalties
Based on FY 81-82
Based o n FY 82-83
Upper
Performance Arcadia Claremont Commerce Culver City Gardena H e r m o s a Beach
La Mirada Long Beach Montebello
Norwalk R e d o n d o Beach Santa M o n i c a SCRTD Torrance Total
Bound *
--
1,050 40,382 31,649 21,261 7,526 18,281 148,129 50,979 167,295 ---
4,920,504 26,981 5,434,037
8,300
1,700 22,650 70,700 103,100 2,300 11,050 546,750 121,700 63,550 1,100 434,750
9,255,900 102,050 10,745,600
Performance --
m 4,031 112,472 ---
m -36,046 69,307 ---
13,687,062 79,428 13,988,346
Upper Bound *
8,400 2,100 23,450 69,300 105,750 1,100
6,350 566,250 127,650 56,300 1,000 517,400
10,141,350 106,400 11,732,800
*The u p p e r b o u n d for e a c h o p e r a t o r is 5% o f its state and federal o p e r a t i n g subsidies. Penalties w e r e to be withdrawn from future funding. Source: Los A n g e l e s C o u n t y Transportation Commission, 1985.
Not only were the operators successful in eliminating the threat of competition, but they also preserved any "unearned b o n u s e s " - penalty reduct i o n s - in the revised incentive pool. These changes reduced the incentive to improve performance: first, by eliminating the threat of competition from private companies, and second, by allowing agencies to recapture the penalty in subsequent years. For example, Long Beach Transit, which earned only $416,390 of their possible bonus of $625,680 in 1986-87, or 66.55%0, was able to regain $111,827 by 1987; 84.42% of their upper bound. SCRTD recaptured $2.23 from their $2.77 million penalty for the same period. Penalties still existed, but they were ineffective. The TPM continued until 1990. Although performance was calculated against standards and penalties and bonuses assessed, there was little incentive for the largest operators to improve their management. A small percentage of "their" money was shifted to smaller operators, but this was insufficient to change managerial behavior. In 1990, six members of the SCRTD Board of Directors became Commissioners of LACTC as a result of another reorganization plan initiated by the State Legislature. As SCRTD produces 86% of the bus service, the Commissioners have indicated that they would be awarding incentives to their own agency and wished to eliminate the incentive program. The Commission, however, must continue to monitor performance in accordance with the 1979 legislation. The incentive program for transit in Pennsylvania had a similar fate (Miller, 1980). Incentives for superior performance were introduced in 1979 and later abandoned. The larger operators were penalized for
deteriorating performance and lobbied against the program. Like the operators in Los Angeles, t h o s e in Pennsylvania are now required to report performance without incentive or penalty assessments. The California TDA audits, which are also required from transit operators in Los Angeles County, have been more influential on managerial behavior. Both management and the board of directors are required to consider changes recommended by the auditor. Proposals are also reviewed by the countywide performance review committee, and progress must be reconsidered in subsequent audits. Public discussion of potential deficiencies has had more influence on managerial behavior than potential loss of a small bonus. When there is a close association between the regulatory agency and regulated operators, it is not uncommon for the operators to succeed by capturing the agency. Economists observing this pattern in federal agencies refer to it as regulatory capture (Stigler, 1971). The LACTC staff began the TPM program armed with a consultant's report and reinforced by a mandate from the Legislature. Commissioners were less enthusiastic; they recognized that improved performance would require reduction of service in low density areas and a lower real growth in salaries and benefits for transit employees. As elected officials, they sought to avoid situations that required them to reward one group at the expense of another; they preferred a solution that gave the appearance of rewarding all groups. Transit managers were opposed; they desired the least interference and lobbied both the staff and commissioners to minimize requirements. The LACTC staff were required to meet fre-
G. J. FIELDING
490
quently with the operators in order to design a performance monitoring and incentive program that would satisfy the operators and yet fulfill the legislative requirement. Interaction within the network tended to gravitate towards a consensus; and, because the outcome was more important to the operators than it was to the L A C T C staff, the views of the operators prevailed. The 1981 T P M program was adopted but never enforced. The staff who helped to design the initial program moved on, some to operating agencies, and their replacements were indifferent to the T P M program and more interested in working with the operators. The T P M program was modified in 1985 and the incentive portion dropped in 1990.
CONCLUSION Although performance-based approaches to management have helped agencies focus on cost control during the 1980s, the problems have not been solved. Far too much effort continues to be devoted pursuing markets in which transit cannot be effective. Performance is improving, however, and performance evaluation programs deserve some of the credit. In the largest agencies, real cost-per-revenue vehicle hour for bus service has increased by less than 1 o70 per annum between 1986 and 1989. These large agencies, with more than 250 vehicles in peak-hour service, operate two-thirds of all bus service. Much still remains to be accomplished through diffusion of successful practices. O f the three case studies, only Californian T D A audits can be regarded as a success. These rely on a small set of diagnostic indicators to reveal progress over three years and then use the information to identify causes for deterioration. Agencies have three years to correct adverse reports. In addition, agencies are required to discuss the report, plus any remedial actions, with the performance committee representing groups which will be affected by the changes. Employee representatives serve on this committee. The incentive program designed for Los Angeles County has been ineffective. Too much effort was required to calculate the indices and decide penalties and rewards. A n d the incentives came too late for any beneficial change. It represented a novel attempt to introduce private incentives that overlooked the complexity of public management. I m p r o v e m e n t in transit performance requires cooperation between labor, management and the governing board rather than conflict. The federal triennial reviews do not go far enough. U n i f o r m data are used, but there is no requirement that a small set of indicators be reported to all agencies. Agencies are not required to act upon recommendations unless the agency has failed to comply with federal regulations. The federal review could be more effective if modified to incorporate features of the Californian audits.
Failure of incentive programs has been a disappointment. The experience in Michigan, Pennsylvania and Los Angeles suggests that failure occurs because allowance is not made for differences in operating policies. Large agencies operate regional service seven clays a week and almost 24 hours a day. Service is slower and more expensive to produce than community-based service operated by municipal systems. And when measured against statewide or regional standards, the regional agency appears less efficient, although they may be transporting many more passengers per service hour. They are penalized for assisting those individuals and areas that transit policy was originally designed to help. Incentive programs have been more helpful within agencies. Coach operators are rewarded for reliable attendance and safe driving. And private contractors have been offered cash incentives if they earn more than expected fare revenue or provide service at a lower cost. But cash incentives are not granted to public managers. For example, managers are not rewarded for reducing costs below budget or operating their maintenance departments efficiently. Public administrators are expected to achieve these economies; to reward them financially might be regarded as a gift of public funds. Employees in public agencies would be more likely to make the difficult decisions required to improve efficiency if they shared in the resulting benefits. New indicators and procedures for measurement would be required, as well as different organizational arrangements. Such requirements provide a potential direction for future research in performance evaluation.
REFERENCES
Caltrans (1982) Transit Performance Audit Guidebook. State Department of Transportation, Sacramento, CA. Dajani J. S., Gilbert G. (1978) Measuring the performance of transit systems. Transpn, Plann. Technol., 4, 2, 97 103. Fielding G. J., Glauthier R. E., Lave C. A. (1978) Performance indicators for transit management. Transportation, 7, 4, 365-379. Fielding G. J., Mundle S. R., Misner J. (1982) Performance based funding allocation guidelines for transit operations in Los Angeles county. Transpn. Res. Rec., 857, 14-18. Fielding G. J., Babitsky T. T., Brenner M. E. (1985) Performance analysis for bus transit. Transpn. Res. A, 19A, 1, 73-82. Fielding G. J. (1987) Managing Public Transit Strategically. Jossey-Bass, San Francisco, CA. Hatry H. P. (1978) The status of productivity measurement in the public sector. Public Admin. Rev., 38, 1, 28-33. Holec J. M., Schwager D. S., Fandalian A. (1980) Use of federal Section 15 data in transit performance: Michigan program. Transpn. Res. Rec., 765, 36-38. Lamare J. L. (1981) Intergovernmental finance, productivity, and the local match question: The case of California's transit subsidy policy. Public Admin. Rev., 41, 4, 463-470. Miller J. H. (1980) The use of performance-based methodologies for the allocation of transit operating funds. Traffic Q., 34, 4, 555-585.
Transit performance evaluation in the U.S.A. Nash C. A. 0978) Management objectives in bus transport. J. Transport Econom. Policy, 12, 1, 70-85. Smerk G. M. (1991) The Federal Role in Urban Mass Transportation. Indiana University Press, Bloomington, IN. Stigler G. (1971) The theory of economic regulation. Bell J. Econom., Spring, 3-21. Talley W. K. (1988) An economic theory of the public transit firm. Transpn. Res., 22B, 1, 45-54. Talley W. K., Becker J. (1982) A single measure for evaluating public transit systems. Transpn. Q., 36, 3,423-431.
491
U.S. Congress, Committee on Public Works and Transportation (1988) Compilation of Selected Surface Transportation Laws: Title 23, United States Code. U.S. Government Printing Office, Washington, DC. U.S. DOT (1991) Section 9 Triennial Reviews. UMTA Notice, N9010.1, Washington, DC. U.S. DOT (1991) Public Transportation in the United States: Performance and Condition. The Secretary of Transportation's Report to the U.S. Congress Pursuant to 49USC308.