- Email: [email protected]
Financial performance analysis of Ontario (Canada) Credit Unions: An application of DEA in the regulatory environment
European Journal of Operational Research 139 (2002) 339–350 www.elsevier.com/locate/dsw
O.R. Applications
Financial performance analysis of Ontario (Canada) Credit Unions: An application of DEA in the regulatory environment Peter Pille a
a,*
, Joseph C. Paradi
b
School of Information Technology Management, Faculty of Business, Ryerson University, 350 Victoria Street, Toronto, Ont., Canada M5B 2K3 b Centre for Management of Technology and Entrepreneurship, University of Toronto, Toronto, Ont., Canada Received 28 February 2001; accepted 2 May 2001
Abstract Models are developed to detect weaknesses in Credit Unions in Ontario, Canada, so that potential financial failures can be predicted. Four data envelopment analysis (DEA) models are presented and compared with the equity to asset ratio, and with the government regulator’s extensively modified ‘‘Z-score’’ model. The equity/asset ratio is shown to provide as good a prediction of failure as any of the other models, and is not improved upon by the much more complex Z-score model. The best DEA model provides results comparable to the equity/asset ratio when a slack adjusted efficiency score is used to measure efficiency, particularly for Credit Unions with larger asset sizes. However, DEA also provides indications of where opportunities lie for improvements by weak units by providing specific information, relevant to managers. Hence, for each Credit Union, comparison is made with a peer group of efficient entities that the inefficient institution’s management can emulate to improve their performance. 2002 Elsevier Science B.V. All rights reserved. Keywords: Data envelopment analysis; Banking; Forecasting; Failure; Efficiency
1. Introduction Canadians are one of the world’s best served citizens from a banking services availability point of view. There are over 8000 bank branches, and
* Corresponding author. Tel.: +1-416-976-5000x7746; fax: +1-416-979-5249. E-mail addresses: [email protected], [email protected] (P. Pille), [email protected] (J.C. Paradi).
thousands of other financial institution branches serving about 30 million people. Aside from bank branches, trust companies, Credit Unions, securities brokers, mutual funds, insurance companies, provincial savings institutions and consumer credit firms make up the financial services industry (FSI) in Canada. Along with the plethora of FSI firms come various levels of government agencies regulating them to ensure investor safety. In this study, we focussed on the Credit Union sector in the largest Canadian Province, Ontario. The work was
0377-2217/02/$ - see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 1 ) 0 0 3 5 9 - 9
340
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
carried out with the cooperation of the regulator, the Deposit Insurance Corporation of Ontario (DICO) whose intent was to predict financial troubles far enough ahead of actual failure so that appropriate measures may be taken to minimise the losses they have to cover. In 1996, Ontario Credit Unions varied greatly in size from a tiny $59,000 up to $910,000,000 in assets (see Table 1). The data for the models are based on unaudited financial reports provided by each Credit Union to DICO each quarter of each year. The data are considered by DICO to be more reliable for larger Credit Unions (over $2 million in assets). Failures from 1992 to 1996 represented 3.2–4.4% of all Credit Unions in each year (see Table 1). Thus, the purpose of this research is to examine the effectiveness of various data envelopment analysis (DEA) models in detecting financial weakness of individual Credit Unions in Ontario, in the years prior to failure. Our hypothesis is:
ciency measure hBCC , and also with a slack adjusted measure hB . Models with and without constraints on the hyperplane multipliers are used, based on interest rate ranges. An additional model makes use of the projection to the closest point on the VRS frontier (L2 -norm), using the hB efficiency measure. The rest of this paper is organised as follows. Section 2 presents a brief discussion on DEA and efficiency measurement with hB . Section 3 deals with the specific DEA models used to do the analysis. Section 4 presents the results of the work and Section 5 offers our analysis of these results. Section 6 contains our conclusions while Appendix A describes the Z-score definition used by the regulator.
Measures generated by the models, in years prior to failure differ for Credit Unions that fail, from those that do not fail.
Services companies continually seek improved methods to measure the performance of their organisations because they are committed to improve efficiency and effectiveness in their operating units. Managers generally regard conventional methods inadequate because they fail the tests of fairness and equitability. They struggle with the existing methodology as they are trying to implement the results obtained from such processes as performance ratios, regression analysis results and the like. DEA has proven itself to be both a theoretically sound framework for performance measurement and an acceptable method by those being measured. DEA has many advantages over traditional methods such as regression analysis, performance ratios, activity based measurements (ABM), etc.
In the process we may also determine which models and which measures from each model, are the most effective in detecting failure. DEA efficiency measures are compared with the modified Z-score used by the DICO and the equity/asset ratio calculated for each unit. The models are compared with actual Credit Union failures over the years from 1992 to 1996, and with the efficiency measures of the models in the years before failure from 1991 to 1995. In the DEA models, projections are to the variable returns to scale (VRS) frontier using the input oriented method, with the use of the traditional DEA effi-
2. Background 2.1. Data envelopment analysis
Table 1 1996 Credit Union assets (less goodwill) in $1000 Canadian four quarters average, for Credit Unions with operations throughout the entire year Asset range
Number of Credit Unions
Mean $
Median $
S.D. $
Under $2000 Over $2000
147 298
766 44,587
634 19,003
552 93,656
Maximum $ 1,997 909,876
Minimum $
Failures 1992–1996
59 2058
44 66
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DEA was developed from a multi-disciplinary effort by engineers, managers and economists and has seen enormous growth during the past two decades. DEA has been characterised as a new way to organise and visualise performance data in organisations. Scientifically speaking, DEA is a non-parametric, fractional linear programming technique that results in a best practice frontier constructed from the best performing units in the sample. Moreover, DEA provides efficiency ratings for the inefficient units as well as indicators of potential improvement opportunities for such units. For cases where there are a sufficient number of operating units, decision making units (DMU), DEA offers a method of analysis that accommodates multiple inputs and outputs at the same time and is units invariant. DEA has been utilised for studies in over 50 different industries and well over 1000 scientific papers have been written about these studies. As there is ample literature on the basic and extended DEA formulations and mathematical representations, we will not repeat these here. The reader is directed to a set of seminal papers on DEA listed in Charnes et al. (1994). For a comprehensive treatment of DEA the reader is referred also to Ali and Seiford (1993), Ali et al. (1995), Banker (1984), Bowlin et al. (1985), Charnes et al. (1989), and Seiford and Thrall (1990). One of the models used in the research of this paper places constraints on the slopes (hyperplane multipliers) of the piece-wise linear efficient frontier; for extensive treatment of constraints in DEA see Thomson et al. (1990), Charnes et al. (1990), Roll and Golany (1993) and Ali and Seiford (1993).
341
oriented efficiency ð61Þ, and / is the output oriented efficiency ðP1Þ. This method can be used regardless of the method of projection to the frontier, whether input oriented, output oriented, closest point on the frontier, maximum sum of slacks, or minimum sum of slacks to a facet. The inefficient DMU is measured relative to the projected point on the frontier in order to derive its efficiency measure. Consider the example in Fig. 1, where inefficient DMU O has been projected to point p on the facet AB of the efficient frontier. A combined overall measure of efficiency could then be taken as Xp =XO YO =Yp . When there is no slack, this value reverts to the same measure as used in a BCC or CCR input oriented approach and the inverse of the BCC or CCR output oriented approach. In addition, it has the advantage that the proportional reduction in inputs or expansion in outputs, and any remaining slack, are combined into one efficiency measure, rather than one input or output radial measure plus a number of slack values. In general, with m input dimensions and s output dimensions, the overall efficiency measure hB is determined by calculating the efficiency in each dimension separately, and then multiplying the mean of the input efficiencies ðhi Þ by the mean of the inverse of the output efficiencies ðho ¼ 1=/Þ: hB ¼ h o h i but with m 1 X xjp =xjo ; hi ¼ m j¼1
ð2:1Þ
ð2:2Þ
2.2. Efficiency measure hB The DEA models used in this research include the input oriented, VRS model. The traditional efficiency measure for these models is the h measure, which does not account for all the inefficiency if there is slack remaining after the proportional reduction of inputs. An alternative approach is to take the product of the input and output oriented components of efficiency measures, by simultaneous input reduction and output expansion. This measure is essentially h=/, where h is the input
Fig. 1. Efficiency as product of input, output efficiencies.
342
ho ¼
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350 s 1X yjo =yjp : s j¼1
ð2:3Þ
In Eq. (2.1) hB 6 1 for input and output oriented BCC and CCR frontiers. This efficiency measure is the same as the BCC or CCR efficiency scores if there is no slack, for the input oriented measures, and for the inverse score for the output oriented measure. If there is slack in inputs or outputs, the hB score is less than the corresponding BCC or CCR input oriented measure, or the inverse of the output oriented measure. Zeros are accommodated in hB for BCC and CCR. If an original input xjo is zero, that dimension can be dropped from the list of inputs, since xip in the projected point must also be zero for that input, if inputs are not allowed to increase in the projection to the frontier. An original output yjo equal to zero presents no problems since it is in the numerator, assuming that outputs are not allowed to decrease to zero when projected to the frontier. However, it is possible for the projected output yjp to be zero if the original value yjo was zero; in this case that output dimension can be dropped from the efficiency calculation. In this research, the hB measure results in better prediction of Credit Union failure than the traditional h measure, referred to as the hBCC measure. The hB measure is required for Model 2 since hBCC cannot be used for the closest point projection.
3. Failure prediction models Before any progress can be made using DEA, the analyst must build one or more DEA models which accurately reflect the actual production process and is agreed to by DMU management. The general form of the model is shown in Fig. 2. Different models utilise different combinations of inputs and outputs as seen from Table 2; the models are numbered from 1 to 6. Models 1 and 2 are identical except that in Model 2, the projection to the DEA frontier is to the closest VRS point (L2 -norm) (Pille, 1998) rather than by
Fig. 2.
radial input reduction. In Model 2, all inputs and outputs are normalised by the average across all Credit Unions. Model 3 is the same as Model 1 except that multiplier constraints are used, based on interest rates, and loans are separated from cash and investments, to allow for different multiplier weights (different rates of return constraints). Inputs and outputs in Model 4 are similar to those used by Barr and Siems (1991), Siems (1992), and Barr et al. (1993), in their research in predicting bank failures. This model is included in order to compare with our Models 1–3. Differences from the inputs actually used by Barr et al. are that inputs: FTE employees, salary expense, premises and fixed assets, and purchased funds were not included as separate inputs. Also, Model 4 includes other income together with interest income as an output. Model 5 is the modified Z-score used by DICO in ranking Credit Unions, as described in Appendix A. Model 6 is the ratio of average equity to average assets in each year. Ten other DEA models were examined in this research, but are not reported here as they do not improve upon Models 1, 2, or 3. In Model 3, the input non-interest expense, and the outputs equity and net interest income + other incomes have the same weight constraints ranging from 0.9 to 1.1. The idea is that these variables are dollar values that should have approximately equal weight, but a slight variation is allowed in order not to force the DEA algorithm to pick a value exactly equal to one, which may cause computational difficulties. The constraints for the input deposits, and the output loans and cash + investments were taken from the observed distribution of interest rates and rates of return for Ontario Credit Unions over the years.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
343
Table 2 Models for failure prediction Variables/methods
Inputs Non-interest expense Deposits Interest expense Outputs Loans Cash and investments Loans, cash and investments Equity Net interest income and other incomes Deposits Interest income and other incomes Multiplier constraints Closest point Input oriented DICO Z-score (see Appendix A) Average equity/average assets ratio
DEA models (VRS)
Non-DEA models
1
2
3
4
5
6
4. Results 4.1. Separating failures from healthy units Fig. 3 shows that the failed Credit Unions tend to have lower scores than healthy Credit Unions in the years prior to failure, and that the difference diminishes the further back in time. It also shows the difference in the scores from one to five years before failure, with the measures hB for Models 1–4, with the Z-score for Model 5, and equity/asset ratio for Model 6. Failures are taken from 1992 to 1996, with scores in years 1991–1995. The downward sloping lines in Fig. 3 represent the difference between the mean measure for the healthy Credit Unions and the mean measure for the failed Credit Unions for each model, in units of standard deviations of the overall score of each measure. Fig. 3 is for Credit Unions with assets greater than $2 million. The results when all Credit Unions are included are similar, but in that case the difference between the failed and healthy Credit Unions is slightly less pronounced (not shown). A one tailed Wilcoxon Rank Sum test is used to determine if there is a statistically significant difference in the distributions of the scores of healthy
and failed units (at a ¼ 0:05). Using Credit Unions of all asset ranges, measure hB for Model 1 shows a statistically significant difference for up to three years before failure. The Z-score (Model 5) and the equity/asset ratio (Model 6) show a statistically significant different for up to five years before failure. Note that in this test, we are comparing the distribution of the scores of a relatively small number of failures, to the scores of a large number of healthy Credit Unions. We need a method of comparing the effectiveness of each model in predicting failure. The proposed approach is to assume that the lower the score from any of the models, the higher the probability of failure will be. Thus the regulatory agency should investigate as many of the lowest scoring Credit Unions as its resources allow. However, for higher scoring Credit Unions, relying on these scores may reduce inspection costs. 4.2. Failure prediction index The results from DEA Model 1 (using hB ) and Model 6 (equity/asset ratio), using failure data in 1995, are used as an example in developing a ‘‘Failure Prediction Index’’ for comparing the
344
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
Fig. 3. Difference of mean scores of non-failures and failures over time, assets > $2 million, Models 1–6.
various models. The higher the Failure Prediction Index, the better the model is in predicting failure. Models 1 and 6 are plotted together in Fig. 4 for comparison. If one model has a line that is consistently above the other, then it is superior in
predicting failure. The perfect model to predict failure would assign the lowest scores to all the failures and its line would be at 100% at each point. Note that Fig. 4 is for failures in one particular year, and with the scores in one year prior to failure. The distributions differ for failures in
Fig. 4. Percent of failures vs. number of failures, one year prior to failure, assets > $2 million.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
other years, and for different years prior to failure, and of course, each model has different patterns. In order to develop the idea of a Failure Prediction Index, consider the hB efficiency scores of Credit Unions placed in increasing sequence, spread over the range between zero and one, from left to right, respectively. At any efficiency score value we select, the poorer performers or failures would tend to be on the left of that score value. Then as we increase the score value we select, the percent of Credit Unions to the left that are failures would tend to decrease, since we are including relatively more of the better performers. If the efficiency score is a good predictor of failure, then the percentage of failures at some increasing score value would suddenly greatly decrease. A graphical way of looking at the Failure Prediction Index is that it is the value of the largest area in Fig. 4 formed by the rectangle with the origin at the bottom left, and a failure found in the graph at the upper right. This area is then normalised by the total number of failures so comparison can be made with other years where the total number of failures is different. For each model, year of failure, and year prior to failure, the Failure Prediction Index is defined as the maximum value of
345
(the vertical axis in Fig. 4). The second ratio is the fraction of the total failures that are found as the score is increased (the horizontal axis in Fig. 4, normalised by the overall total number of failures). We would like to have both of these factors as high as possible. In Fig. 4, 11 of the 297 Credit Unions failed, with most of the failures having relatively low scores in 1994 ðhB from DEA, or the equity/asset ratio). The maximum Failure Prediction Index for Model 1 in Fig. 4 occurs at the seventh failure, where 9 Credit Unions had equal or lower hB scores (7 failures and 2 healthy), or about 78% of Credit Unions failed, so the index is 7=9 7=11 100 ¼ 49:5%. The hB score of the seventh failure is 0.58; at the eighth failure, the hB value is very close at 0.59, but includes three more healthy units. For Model 6, the Failure Prediction Index is obtained at the eighth failure, where 11 Credit Unions had equal or lower equity/asset ratios than the eighth failure, for an index value of 8=11 8=11 ¼ 52:9%. The equity/asset ratio at the eighth failure is 1.0%. The last three failures (9, 10, and 11) in Fig. 4 failed due to special circumstances. At failure 9, the hB score for Model 1 is 0.72, and for Model 6, the equity/asset ratio is 3.6%. The results shown in Fig. 4 for one year prior to failure are for only two models, and only for failures in 1995. Table 3 shows the average of the Failure Prediction Index for failures in other years and models. A separate average of the Failure Prediction Index is provided for each of five years prior to failure. For example, in Model 1, and for
number of failures 6 score total number of Credit Unions 6 score number of failures 6 score 100: total number of failures The first ratio above is the fraction of Credit Unions that are failures as the score is increased
Table 3 Mean Failure Prediction Index and standard deviation for years prior to failure Model
1 2 3 4 5 6
Mean Failure Prediction Index for years prior to failure
Standard deviation of Failure Prediction Index for years prior to failure
1 year
2 years
3 years
4 years
5 years
1 year
2 years
3 years
4 years
5 years
38.1 31.0 32.0 6.6 27.6 34.8
15.1 14.1 10.4 7.2 16.4 19.9
14.2 10.5 8.7 6.7 12.4 15.2
12.4 11.1 11.3 4.2 12.7 14.9
5.8 6.7 4.2 3.8 7.7 14.1
19.6 13.0 16.6 1.7 12.2 13.6
4.3 2.6 3.0 3.7 9.6 4.2
7.0 7.9 2.2 5.1 3.8 4.2
7.0 6.4 1.8 0.5 5.3 10.4
0.0 0.0 0.0 0.0 0.0 0.0
Based on hB for the DEA models. Assets > $2 million (1996 $).
346
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
Table 4 DEA Model 1 peer group comparison for a weak Credit Union (failure in 1995, 1994 data)
Weak Credit Union Peer Credit Unions (efficient)
Credit Union
Assets (1996 $ millions)
Equity/assets (%)
k (% of peer)
L N M O
138.3 104.2 79.6 306.4
0.5 7.1 7.9 6.4
2 72 26
one year prior to failure, five cases were averaged, for failures in each of years 1992–1996. This gives an average of 38.1 for Model 1, for one year prior to failure (from the five values 50.6, 57.1, 14.8, 49.5, and 19.1 for scores in 1991 to 1995, respectively). Model 1 shows greater success (higher index) one year prior to failure in 1991, 1992 and 1994 than in 1993 and 1995. Table 3 also shows the standard deviation of the Failure Prediction Index for each year prior to failure. For five years prior to failure, only one year of data is used, resulting in zero for the standard deviation. Since Z-score data (Model 5) were available for only the years 1991–1996, comparisons are not made prior to 1991. Wheelock and Wilson (1995) consider banks in the US with an equity/asset ratio less than 2% to be a failure. The cut-off values for equity/assets scores from the Failure Prediction Index range from )2.0% to 1.7% for different years of failure, which are consistent with Wheelock and Wilson (1995). 4.3. Peer group comparison In this section a comparison is made between a weak Credit Union and its peer group as identified by DEA Model 1. The peer group represents the efficient Credit Unions on the DEA frontier that form a facet or part of a facet that the inefficient Credit Union is projected to, by the input oriented approach. This comparison permits identification of the deficiencies of the weak Credit Union with respect to its inputs and outputs in the model. The target inputs and outputs of the weak Credit Union are taken as a convex combination of the inputs and outputs of the peers.
hBCC
hB
0.92 1 1 1
0.59 1 1 1
The eighth failure in Fig. 4 is Credit Union L, which was identified by DICO as weak, and was purchased by Credit Union P. The purchase took place in 1995, and the comparison is made for the prior year 1994. Table 4 shows that the peers for Credit Union L (M, N, and O) are of a similar size, and that the equity/asset ratio of Credit Union L is much less than that of the peers. The k values total 100%, and indicate the percentage of each peer’s inputs and outputs that represent an efficient target for charter L. Of the three peers, the target for Credit Union L is most similar to Credit Union M ðk ¼ 72%Þ. The hBCC value of 0.92 for L indicates that all inputs for Credit Union L need to be reduced by 8% to reach the target. After this proportionate input reduction, further reductions are required if there is any remaining input slack, but this is not the case for Credit Union L. An increase in outputs (output slack) to reach the target is required (in this case, primarily an increase in equity, from $0.7 million to $9.8 million). The hB value indicates a slack adjusted efficiency measure of 0.59 for Credit Union L, compared to 1.0 for each peer.
5. Analysis Reasons for the variability in the Failure Prediction Index deserves further discussion. The methodology used in this research considers failures during any point in a calendar year to be failures in the same year. Thus, failures can be up to 11 months apart, yet they are considered as failures at the same point in time. The year of failure can be arbitrary if the date of failure is assigned to December rather than to January of the next year. The categorisation of a Credit Union as a failure by
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DICO to a specific date may not always be meaningful if a Credit Union has been faltering for some time. Performing the analysis on a quarterly basis would be more accurate. Credit Unions are considered a failure when DICO assigns a Loss Code, or considered to be the weak partner in a merger. Mergers ‘‘encouraged’’ by DICO in order to save a weak Credit Union, in all probability, have not all been identified and thus are not categorised as failures. Catastrophic failures such as a failure of an employer Credit Union due to a plant closure, and failure due to fraud, are not predictable with Models 1–6 (or perhaps by any other method). DICO management indicates that failures due to mismanagement are often believed to be due to fraud, but cannot be proven. The number of values of the Failure Prediction Index that is averaged in Table 3 varies from five values for one year prior to failure to one value for five years prior to failure. This small number of values makes it difficult to establish whether there is a statistically significant difference between the models, especially beyond one year prior to failure. The non-parametric Wilcoxon Rank Sum test indicates that for a two tailed test with a ¼ 0:10, we cannot reject the hypothesis that the distribution of the Failure Prediction Index is the same for Models 1, 2, 3, 5 or 6 for one year prior to failure. Only for Model 4 is there a statistically significant difference in the distribution with the other models, at a ¼ 0:05, for one year prior to failure. At three years prior to failure, we have only three data points to compare for the different models for the Failure Prediction Index; in this case, the only models that can be shown to be statistically significantly different are between Models 3 and 6, and between Models 4 and 6, at a ¼ 0:10. Thus, for Credit Unions with assets larger than $2 million dollars (1996 $) there does not appear to be a clearly best model for prediction of failure between the DEA Models 1, 2 and 3, the Z-score Model 5, and the equity/asset ratio in Model 6. DEA Model 4 does not fare well in comparison. Placing constraints on the hyperplane multipliers in Model 3 does not improve the results from Model 1; nor does projecting inefficient Credit Unions to the closest point on the frontier, as in
347
Model 2. For one year prior to failure, Model 1 has a higher mean Failure Prediction Index than the others, but also a higher variation. For two years prior to failure, Model 6 has the highest mean Failure Prediction Index and also a lower variation than Model 5, which has the next highest mean Failure Prediction Index. For three years prior to failure, Model 6 has the highest mean Failure Prediction Index followed closely by Model 1, with a similar variation. Overall, Model 6 with the simple ratio of equity/assets appears as good as any of the other models. However, while this measure is considered to be a universal indicator of financial health, it does not contribute any insights into the reasons for the problems, nor offers direction to management as to how to address them. Moreover, we must consider that the equity in a Credit Union comes from contributed capital by the members together with retained earnings. Since the very essence of these organisations is to provide the lowest cost lending and are essentially non-profit ones, not much capital is accumulated. Assets are, for the most part, loans to members or investments. It is assumed that loan losses are rare because the members’ deposits are at stake and the borrowers are closely tied to the community served by the Credit Union. All this adds up to unusually small equity/asset ratios even for healthy units when compared to other financial institutions. Therefore, Credit Unions can operate for years with a dangerously low equity/asset ratio because as long as loan defaults are low, they stay solvent. The importance of this background is that this makes the equity/asset ratio all important because a unit with a very low figure can fail relatively easily but it can avoid failure for years. Therefore, the ratio may appear to be a predicting failure far in advance of the actual problem but, in reality, it just states the fact that low equity/asset ratio can lead to problems. DEA Model 1 offers significant benefits to DICO as the regulator attempts to head off problems at suspect Credit Unions. By offering timely advice to Credit Union managers as to how to improve operations, ultimate failure may be prevented because managerially useful information can be provided to fix operating characteris-
348
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
tics other than the equity/asset ratio. Moreover, DEA provides useful information for inefficient Credit Unions that do not have low equity/asset ratios and management can make adjustments to avoid problems that may, in the long run, lead to a depletion of their equity. After all, equity erosion can be the result of operational losses as well as bad loans. Finally, if non-member targeted investments are made unwisely, the Credit Union may be in serious trouble if such investments should go bad. A vivid example of this can be seen in the US based Savings and Loan debacle of the past decade. Each of the models shows a higher mean Failure Prediction Index when DMUs analysed are restricted to those over $2 million dollars in asset size (1996 $), and slightly higher again when the restriction is placed at over $5 million dollars. Similarly, when Credit Unions that fail due to plant closure or fraud are not included, each of the models shows a higher mean Failure Prediction Index. The DEA models (1, 3, and 4) show higher values for the mean Failure Prediction Index when the slack adjusted hB measure is used than when the traditional hBCC measure is used (DEA Model 2 does not use the traditional hBCC measure).
6. Conclusions From the models included in this research, the equity/asset ratio and some DEA models appear to be equally competent in predicting the failure of Credit Unions. However, DEA Model 1 offers indicators of where the problems are and how to address them. Hence it should be the preferred tool for the regulator. Each of the models shows that failures, on an average, have lower scores than healthy units, for up to three years before failure, thus our hypothesis is proven. Prediction of failure is most reliable at one year prior to failure, and declines as we go further out. Prediction improves when only larger asset sized DMUs are included, and also when failures due to plant closure or fraud are excluded. Catastrophic failures due to the latter two causes cannot be predicted and should be excluded from all
analyses. DICO management believes that many cases of mismanagement are actually fraud but that cannot be proven. If this belief is true, then prediction of failure is more difficult than it would otherwise be. The models in this work do not consider the risk involved if a Credit Union has a large proportion of its assets in a single large loan or investment. Yet, this may be the most serious potential problem because a large loan default may well wipe out the entire equity of the Credit Union. Hence, size matters because the relative size between the firm’s equity and the largest loan or investment is a crucial survival issue. Improved results are expected if the models were run on a quarterly basis, rather than annually, as in this research. The availability of audited data would also be of benefit, although not likely to be available on a quarterly basis. The number of failures in any year as a percentage of the total number of Credit Unions is small. Prediction of failure of this small set of Credit Unions produces highly variable results for failures in specific years or certain years prior to failure. Attempting to apply a fixed cut-off score value to predicting failure could lead to high error rates in many instances. The equity/asset ratio provides as good a prediction of failure as any of the other methods, and validates the importance placed on this ratio by DICO. DICO’s Z-score does not appear to be a better tool than the equity/asset ratio. DEA Model 1 (with no multiplier constraints) provides results comparable to the equity/asset ratio when the slack adjusted hB measure is used, particularly for larger asset sizes. The use of the traditional input oriented measure ðhBCC Þ does not provide results as good as the results given by the slack adjusted hB measure. In this research, the use of multiplier constraints or projection to the closest point on the frontier, do not improve the DEA results. DEA provides prediction of failure comparable to or better than the DICO’s Z-score (particularly for larger Credit Unions) without the necessity of determining a set of arbitrary (or empirically derived) weights to improve the results. Moreover, DEA provides guidance to Credit Union management for improvements, the Z-score does not.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DEA Model 4, based on (but not identical to) the variables used by Barr and Siems (1991), Siems (1992), and Barr et al. (1993), does not perform as well as the other models analysed in this work. It should be noted that there are perhaps significant differences between the Model 4 used here and the original used by Barr et al. In this research, the VRS DEA model is used due to the wide range of asset sizes of Ontario Credit Unions. In the Barr et al. study, the CRS model was used, and the asset sizes were much more restricted. They did not use DEA results alone to predict the failure of US banks instead, DEA results were combined with the CAMEL rating to improve failure prediction. The benefits that DEA (such as Model 1) provides can be classified into three areas: • DEA does not require weights to be determined in combining ratios for ranking of Credit Unions. • DEA provides peer groups (relatively efficient Credit Unions) that inefficient units can compare themselves to, in order to improve their management practices. DEA also provides the amounts by which inputs should be reduced, and outputs might be increased, in order to become efficient. The regulator may use these results to monitor change in the units. • The combination of the DEA results in Model 1 with DICO’s Z-score or the equity/asset ratio can be used to identify those Credit Unions that might become a problem. A low score in any one model might not indicate a serious problem in the Credit Union under examination, however, a low score in any two or all of them, or a steady decline over successive quarters, should alert DICO to investigate further.
7. Concluding remarks This study forms a part of a continuing research program on productivity measurement technology in the Canadian financial services industry by the researchers at the Centre for Management for Technology and Entrepreneurship at the University of Toronto. Past studies include the examination of both retail and commercial branch
349
productivity in banks, Schaffnit et al. (1997); historical progress of the largest six banks over a 15 year period; Property and Casualty industry; and the Mutual Funds industry (papers are being prepared from all of these studies). During this particular work we focussed on the needs of the regulator. One of the most important outcomes is the verification of the results with the regulator’s own methodology, thus validating these findings.
Acknowledgements The authors would like to thank Kwabena A., Charles M., Bill F., Rudy B. and Francois G. for their invaluable support and their valuable time given so freely. This research was made possible by the generous financial support of the Toronto Dominion Bank, The Royal Bank of Canada, Bell Canada and the funding agencies of SSHRC and NSERC.
Appendix A. DICO Z-score A ranking of the performance of each Credit Union by DICO is performed in each quarter. A Z-score is calculated based on the following financial data, with the relative weight shown in brackets: Capital adequacy ¼ Regulatory capital /net assets (15) • assets are adjusted for imputed interest expense and taxes, • capital is reduced based on the likelihood of a sudden change in interest rates (shock test). Net interest income/average assets (7). Liquid assets/(deposits + borrowings) (1). Non-interest expense/average assets (6). Credit risk ¼ Loans delinquent over 30 days /risk weighted loans (8). From each ratio an individual Z-score is computed as (ratio ) mean)/standard deviation. The calculation for the standard deviation contains weighting based on the asset size of each individual Credit Union. Then the five Z-scores are weighted
350
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
(the values in brackets above) and added together (the last two with negative sign). The variables Regulatory Capital and Equity are similar, and are used in different models. Regulatory Capital includes the effect of DICO’s ‘‘shock test’’, or effect of a sudden increase in interest rates. Also, Regulatory Capital is taken at the end of the fourth quarter and is not averaged over the year. Equity, as used in this research, is the average of the quarterly values, with adjustments for imputed interest expense (interest payments by a Credit Union are imputed in those cases where no interest payments, or very low values, are reported).
References Ali, A.I., Seiford, L.M., 1993. The mathematical programming approach to efficiency analysis. In: Fried, H.O., Lovell, K.C.A., Schmidt, S.S. (Eds.), The Measurement of Productive Efficiency. Oxford University Press, Oxford, NY. Ali, A.I., Lerme, C.S., Seiford, L.M., 1995. Components of efficiency evaluation in data envelopment analysis. European Journal of Operational Research 80, 462–473. Banker, R.D., 1984. Estimating most productive scale size using data envelopment analysis. European Journal of Operations Research 17 (1), 35–44. Barr, R.S., Siems, T.F., 1991. Predicting bank failure using DEA to quantify management quality. Research Paper, Federal Reserve Bank of Dallas. Barr, R.S., Seiford, L.M., Siems, T.F., 1993. An envelopmentanalysis approach to measuring the managerial efficiency of banks. Annals of Operations Research.
Bowlin, W.F., Charnes, A., Cooper, W.W., Sherman, H.D., 1985. Data envelopment analysis and regression approaches to efficiency estimation and evaluation. Annals of Operations Research 1 (2), 113–138. Charnes, A., Cooper, W.W., Wei, Q.L., Huang, Z.M., 1989. Cone ratio data envelopment analysis and multi-objective programming. International Journal of Systems Science 20 (7), 1099–1118. Charnes, A., Cooper, W.W., Huang, Z.M., Sun, D.B., 1990. Polyhedral cone-ratio DEA models with an illustrative application to large commercial banks. Journal of Econometrics 46, 73–91. Charnes, A., Cooper, W.W., Lewin, A.Y., Seiford, L.M., 1994. Data Envelopment Analysis: Theory, Methodology, and Application. Kluwer Academic Press, Dordrecht. Pille, P., 1998. Performance Analysis of Ontario Credit Unions, PhD dissertation, University of Toronto. Roll, Y., Golany, B., 1993. Alternate methods of treating factor weights in DEA. OMEGA International Journal of Management Science 21 (1), 99–109. Schaffnit, C., Rosen, D., Paradi, J.C., 1997. Best practice analysis of bank branches: An application of DEA in a large Canadian bank. European Journal of Operational Research 98 (1997), 269–289. Seiford, L.M., Thrall, R.M., 1990. Recent developments in DEA: The mathematical programming approach to frontier analysis. Journal of Econometrics 1–2 (46), 7–38. Siems, T.F., 1992. Quantifying management’s role in bank survival. Federal Reserve Bank of Dallas – Economic Review, January. Thomson, R.G., Langemeier, L.N., Lee, C.T., Thrall, R.M., 1990. The role of multiplier bounds in efficiency analysis with application to Kansas farming. Journal of Econometrics 46, 93–108. Wheelock, D.C., Wilson, P.W., 1995. Why do banks disappear? The determinants of U.S. bank failures and acquisitions. Federal Reserve Bank of St. Louis, Research Division Working Papers 95-013A.
O.R. Applications
Financial performance analysis of Ontario (Canada) Credit Unions: An application of DEA in the regulatory environment Peter Pille a
a,*
, Joseph C. Paradi
b
School of Information Technology Management, Faculty of Business, Ryerson University, 350 Victoria Street, Toronto, Ont., Canada M5B 2K3 b Centre for Management of Technology and Entrepreneurship, University of Toronto, Toronto, Ont., Canada Received 28 February 2001; accepted 2 May 2001
Abstract Models are developed to detect weaknesses in Credit Unions in Ontario, Canada, so that potential financial failures can be predicted. Four data envelopment analysis (DEA) models are presented and compared with the equity to asset ratio, and with the government regulator’s extensively modified ‘‘Z-score’’ model. The equity/asset ratio is shown to provide as good a prediction of failure as any of the other models, and is not improved upon by the much more complex Z-score model. The best DEA model provides results comparable to the equity/asset ratio when a slack adjusted efficiency score is used to measure efficiency, particularly for Credit Unions with larger asset sizes. However, DEA also provides indications of where opportunities lie for improvements by weak units by providing specific information, relevant to managers. Hence, for each Credit Union, comparison is made with a peer group of efficient entities that the inefficient institution’s management can emulate to improve their performance. 2002 Elsevier Science B.V. All rights reserved. Keywords: Data envelopment analysis; Banking; Forecasting; Failure; Efficiency
1. Introduction Canadians are one of the world’s best served citizens from a banking services availability point of view. There are over 8000 bank branches, and
* Corresponding author. Tel.: +1-416-976-5000x7746; fax: +1-416-979-5249. E-mail addresses: [email protected], [email protected] (P. Pille), [email protected] (J.C. Paradi).
thousands of other financial institution branches serving about 30 million people. Aside from bank branches, trust companies, Credit Unions, securities brokers, mutual funds, insurance companies, provincial savings institutions and consumer credit firms make up the financial services industry (FSI) in Canada. Along with the plethora of FSI firms come various levels of government agencies regulating them to ensure investor safety. In this study, we focussed on the Credit Union sector in the largest Canadian Province, Ontario. The work was
0377-2217/02/$ - see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 1 ) 0 0 3 5 9 - 9
340
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
carried out with the cooperation of the regulator, the Deposit Insurance Corporation of Ontario (DICO) whose intent was to predict financial troubles far enough ahead of actual failure so that appropriate measures may be taken to minimise the losses they have to cover. In 1996, Ontario Credit Unions varied greatly in size from a tiny $59,000 up to $910,000,000 in assets (see Table 1). The data for the models are based on unaudited financial reports provided by each Credit Union to DICO each quarter of each year. The data are considered by DICO to be more reliable for larger Credit Unions (over $2 million in assets). Failures from 1992 to 1996 represented 3.2–4.4% of all Credit Unions in each year (see Table 1). Thus, the purpose of this research is to examine the effectiveness of various data envelopment analysis (DEA) models in detecting financial weakness of individual Credit Unions in Ontario, in the years prior to failure. Our hypothesis is:
ciency measure hBCC , and also with a slack adjusted measure hB . Models with and without constraints on the hyperplane multipliers are used, based on interest rate ranges. An additional model makes use of the projection to the closest point on the VRS frontier (L2 -norm), using the hB efficiency measure. The rest of this paper is organised as follows. Section 2 presents a brief discussion on DEA and efficiency measurement with hB . Section 3 deals with the specific DEA models used to do the analysis. Section 4 presents the results of the work and Section 5 offers our analysis of these results. Section 6 contains our conclusions while Appendix A describes the Z-score definition used by the regulator.
Measures generated by the models, in years prior to failure differ for Credit Unions that fail, from those that do not fail.
Services companies continually seek improved methods to measure the performance of their organisations because they are committed to improve efficiency and effectiveness in their operating units. Managers generally regard conventional methods inadequate because they fail the tests of fairness and equitability. They struggle with the existing methodology as they are trying to implement the results obtained from such processes as performance ratios, regression analysis results and the like. DEA has proven itself to be both a theoretically sound framework for performance measurement and an acceptable method by those being measured. DEA has many advantages over traditional methods such as regression analysis, performance ratios, activity based measurements (ABM), etc.
In the process we may also determine which models and which measures from each model, are the most effective in detecting failure. DEA efficiency measures are compared with the modified Z-score used by the DICO and the equity/asset ratio calculated for each unit. The models are compared with actual Credit Union failures over the years from 1992 to 1996, and with the efficiency measures of the models in the years before failure from 1991 to 1995. In the DEA models, projections are to the variable returns to scale (VRS) frontier using the input oriented method, with the use of the traditional DEA effi-
2. Background 2.1. Data envelopment analysis
Table 1 1996 Credit Union assets (less goodwill) in $1000 Canadian four quarters average, for Credit Unions with operations throughout the entire year Asset range
Number of Credit Unions
Mean $
Median $
S.D. $
Under $2000 Over $2000
147 298
766 44,587
634 19,003
552 93,656
Maximum $ 1,997 909,876
Minimum $
Failures 1992–1996
59 2058
44 66
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DEA was developed from a multi-disciplinary effort by engineers, managers and economists and has seen enormous growth during the past two decades. DEA has been characterised as a new way to organise and visualise performance data in organisations. Scientifically speaking, DEA is a non-parametric, fractional linear programming technique that results in a best practice frontier constructed from the best performing units in the sample. Moreover, DEA provides efficiency ratings for the inefficient units as well as indicators of potential improvement opportunities for such units. For cases where there are a sufficient number of operating units, decision making units (DMU), DEA offers a method of analysis that accommodates multiple inputs and outputs at the same time and is units invariant. DEA has been utilised for studies in over 50 different industries and well over 1000 scientific papers have been written about these studies. As there is ample literature on the basic and extended DEA formulations and mathematical representations, we will not repeat these here. The reader is directed to a set of seminal papers on DEA listed in Charnes et al. (1994). For a comprehensive treatment of DEA the reader is referred also to Ali and Seiford (1993), Ali et al. (1995), Banker (1984), Bowlin et al. (1985), Charnes et al. (1989), and Seiford and Thrall (1990). One of the models used in the research of this paper places constraints on the slopes (hyperplane multipliers) of the piece-wise linear efficient frontier; for extensive treatment of constraints in DEA see Thomson et al. (1990), Charnes et al. (1990), Roll and Golany (1993) and Ali and Seiford (1993).
341
oriented efficiency ð61Þ, and / is the output oriented efficiency ðP1Þ. This method can be used regardless of the method of projection to the frontier, whether input oriented, output oriented, closest point on the frontier, maximum sum of slacks, or minimum sum of slacks to a facet. The inefficient DMU is measured relative to the projected point on the frontier in order to derive its efficiency measure. Consider the example in Fig. 1, where inefficient DMU O has been projected to point p on the facet AB of the efficient frontier. A combined overall measure of efficiency could then be taken as Xp =XO YO =Yp . When there is no slack, this value reverts to the same measure as used in a BCC or CCR input oriented approach and the inverse of the BCC or CCR output oriented approach. In addition, it has the advantage that the proportional reduction in inputs or expansion in outputs, and any remaining slack, are combined into one efficiency measure, rather than one input or output radial measure plus a number of slack values. In general, with m input dimensions and s output dimensions, the overall efficiency measure hB is determined by calculating the efficiency in each dimension separately, and then multiplying the mean of the input efficiencies ðhi Þ by the mean of the inverse of the output efficiencies ðho ¼ 1=/Þ: hB ¼ h o h i but with m 1 X xjp =xjo ; hi ¼ m j¼1
ð2:1Þ
ð2:2Þ
2.2. Efficiency measure hB The DEA models used in this research include the input oriented, VRS model. The traditional efficiency measure for these models is the h measure, which does not account for all the inefficiency if there is slack remaining after the proportional reduction of inputs. An alternative approach is to take the product of the input and output oriented components of efficiency measures, by simultaneous input reduction and output expansion. This measure is essentially h=/, where h is the input
Fig. 1. Efficiency as product of input, output efficiencies.
342
ho ¼
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350 s 1X yjo =yjp : s j¼1
ð2:3Þ
In Eq. (2.1) hB 6 1 for input and output oriented BCC and CCR frontiers. This efficiency measure is the same as the BCC or CCR efficiency scores if there is no slack, for the input oriented measures, and for the inverse score for the output oriented measure. If there is slack in inputs or outputs, the hB score is less than the corresponding BCC or CCR input oriented measure, or the inverse of the output oriented measure. Zeros are accommodated in hB for BCC and CCR. If an original input xjo is zero, that dimension can be dropped from the list of inputs, since xip in the projected point must also be zero for that input, if inputs are not allowed to increase in the projection to the frontier. An original output yjo equal to zero presents no problems since it is in the numerator, assuming that outputs are not allowed to decrease to zero when projected to the frontier. However, it is possible for the projected output yjp to be zero if the original value yjo was zero; in this case that output dimension can be dropped from the efficiency calculation. In this research, the hB measure results in better prediction of Credit Union failure than the traditional h measure, referred to as the hBCC measure. The hB measure is required for Model 2 since hBCC cannot be used for the closest point projection.
3. Failure prediction models Before any progress can be made using DEA, the analyst must build one or more DEA models which accurately reflect the actual production process and is agreed to by DMU management. The general form of the model is shown in Fig. 2. Different models utilise different combinations of inputs and outputs as seen from Table 2; the models are numbered from 1 to 6. Models 1 and 2 are identical except that in Model 2, the projection to the DEA frontier is to the closest VRS point (L2 -norm) (Pille, 1998) rather than by
Fig. 2.
radial input reduction. In Model 2, all inputs and outputs are normalised by the average across all Credit Unions. Model 3 is the same as Model 1 except that multiplier constraints are used, based on interest rates, and loans are separated from cash and investments, to allow for different multiplier weights (different rates of return constraints). Inputs and outputs in Model 4 are similar to those used by Barr and Siems (1991), Siems (1992), and Barr et al. (1993), in their research in predicting bank failures. This model is included in order to compare with our Models 1–3. Differences from the inputs actually used by Barr et al. are that inputs: FTE employees, salary expense, premises and fixed assets, and purchased funds were not included as separate inputs. Also, Model 4 includes other income together with interest income as an output. Model 5 is the modified Z-score used by DICO in ranking Credit Unions, as described in Appendix A. Model 6 is the ratio of average equity to average assets in each year. Ten other DEA models were examined in this research, but are not reported here as they do not improve upon Models 1, 2, or 3. In Model 3, the input non-interest expense, and the outputs equity and net interest income + other incomes have the same weight constraints ranging from 0.9 to 1.1. The idea is that these variables are dollar values that should have approximately equal weight, but a slight variation is allowed in order not to force the DEA algorithm to pick a value exactly equal to one, which may cause computational difficulties. The constraints for the input deposits, and the output loans and cash + investments were taken from the observed distribution of interest rates and rates of return for Ontario Credit Unions over the years.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
343
Table 2 Models for failure prediction Variables/methods
Inputs Non-interest expense Deposits Interest expense Outputs Loans Cash and investments Loans, cash and investments Equity Net interest income and other incomes Deposits Interest income and other incomes Multiplier constraints Closest point Input oriented DICO Z-score (see Appendix A) Average equity/average assets ratio
DEA models (VRS)
Non-DEA models
1
2
3
4
5
6
4. Results 4.1. Separating failures from healthy units Fig. 3 shows that the failed Credit Unions tend to have lower scores than healthy Credit Unions in the years prior to failure, and that the difference diminishes the further back in time. It also shows the difference in the scores from one to five years before failure, with the measures hB for Models 1–4, with the Z-score for Model 5, and equity/asset ratio for Model 6. Failures are taken from 1992 to 1996, with scores in years 1991–1995. The downward sloping lines in Fig. 3 represent the difference between the mean measure for the healthy Credit Unions and the mean measure for the failed Credit Unions for each model, in units of standard deviations of the overall score of each measure. Fig. 3 is for Credit Unions with assets greater than $2 million. The results when all Credit Unions are included are similar, but in that case the difference between the failed and healthy Credit Unions is slightly less pronounced (not shown). A one tailed Wilcoxon Rank Sum test is used to determine if there is a statistically significant difference in the distributions of the scores of healthy
and failed units (at a ¼ 0:05). Using Credit Unions of all asset ranges, measure hB for Model 1 shows a statistically significant difference for up to three years before failure. The Z-score (Model 5) and the equity/asset ratio (Model 6) show a statistically significant different for up to five years before failure. Note that in this test, we are comparing the distribution of the scores of a relatively small number of failures, to the scores of a large number of healthy Credit Unions. We need a method of comparing the effectiveness of each model in predicting failure. The proposed approach is to assume that the lower the score from any of the models, the higher the probability of failure will be. Thus the regulatory agency should investigate as many of the lowest scoring Credit Unions as its resources allow. However, for higher scoring Credit Unions, relying on these scores may reduce inspection costs. 4.2. Failure prediction index The results from DEA Model 1 (using hB ) and Model 6 (equity/asset ratio), using failure data in 1995, are used as an example in developing a ‘‘Failure Prediction Index’’ for comparing the
344
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
Fig. 3. Difference of mean scores of non-failures and failures over time, assets > $2 million, Models 1–6.
various models. The higher the Failure Prediction Index, the better the model is in predicting failure. Models 1 and 6 are plotted together in Fig. 4 for comparison. If one model has a line that is consistently above the other, then it is superior in
predicting failure. The perfect model to predict failure would assign the lowest scores to all the failures and its line would be at 100% at each point. Note that Fig. 4 is for failures in one particular year, and with the scores in one year prior to failure. The distributions differ for failures in
Fig. 4. Percent of failures vs. number of failures, one year prior to failure, assets > $2 million.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
other years, and for different years prior to failure, and of course, each model has different patterns. In order to develop the idea of a Failure Prediction Index, consider the hB efficiency scores of Credit Unions placed in increasing sequence, spread over the range between zero and one, from left to right, respectively. At any efficiency score value we select, the poorer performers or failures would tend to be on the left of that score value. Then as we increase the score value we select, the percent of Credit Unions to the left that are failures would tend to decrease, since we are including relatively more of the better performers. If the efficiency score is a good predictor of failure, then the percentage of failures at some increasing score value would suddenly greatly decrease. A graphical way of looking at the Failure Prediction Index is that it is the value of the largest area in Fig. 4 formed by the rectangle with the origin at the bottom left, and a failure found in the graph at the upper right. This area is then normalised by the total number of failures so comparison can be made with other years where the total number of failures is different. For each model, year of failure, and year prior to failure, the Failure Prediction Index is defined as the maximum value of
345
(the vertical axis in Fig. 4). The second ratio is the fraction of the total failures that are found as the score is increased (the horizontal axis in Fig. 4, normalised by the overall total number of failures). We would like to have both of these factors as high as possible. In Fig. 4, 11 of the 297 Credit Unions failed, with most of the failures having relatively low scores in 1994 ðhB from DEA, or the equity/asset ratio). The maximum Failure Prediction Index for Model 1 in Fig. 4 occurs at the seventh failure, where 9 Credit Unions had equal or lower hB scores (7 failures and 2 healthy), or about 78% of Credit Unions failed, so the index is 7=9 7=11 100 ¼ 49:5%. The hB score of the seventh failure is 0.58; at the eighth failure, the hB value is very close at 0.59, but includes three more healthy units. For Model 6, the Failure Prediction Index is obtained at the eighth failure, where 11 Credit Unions had equal or lower equity/asset ratios than the eighth failure, for an index value of 8=11 8=11 ¼ 52:9%. The equity/asset ratio at the eighth failure is 1.0%. The last three failures (9, 10, and 11) in Fig. 4 failed due to special circumstances. At failure 9, the hB score for Model 1 is 0.72, and for Model 6, the equity/asset ratio is 3.6%. The results shown in Fig. 4 for one year prior to failure are for only two models, and only for failures in 1995. Table 3 shows the average of the Failure Prediction Index for failures in other years and models. A separate average of the Failure Prediction Index is provided for each of five years prior to failure. For example, in Model 1, and for
number of failures 6 score total number of Credit Unions 6 score number of failures 6 score 100: total number of failures The first ratio above is the fraction of Credit Unions that are failures as the score is increased
Table 3 Mean Failure Prediction Index and standard deviation for years prior to failure Model
1 2 3 4 5 6
Mean Failure Prediction Index for years prior to failure
Standard deviation of Failure Prediction Index for years prior to failure
1 year
2 years
3 years
4 years
5 years
1 year
2 years
3 years
4 years
5 years
38.1 31.0 32.0 6.6 27.6 34.8
15.1 14.1 10.4 7.2 16.4 19.9
14.2 10.5 8.7 6.7 12.4 15.2
12.4 11.1 11.3 4.2 12.7 14.9
5.8 6.7 4.2 3.8 7.7 14.1
19.6 13.0 16.6 1.7 12.2 13.6
4.3 2.6 3.0 3.7 9.6 4.2
7.0 7.9 2.2 5.1 3.8 4.2
7.0 6.4 1.8 0.5 5.3 10.4
0.0 0.0 0.0 0.0 0.0 0.0
Based on hB for the DEA models. Assets > $2 million (1996 $).
346
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
Table 4 DEA Model 1 peer group comparison for a weak Credit Union (failure in 1995, 1994 data)
Weak Credit Union Peer Credit Unions (efficient)
Credit Union
Assets (1996 $ millions)
Equity/assets (%)
k (% of peer)
L N M O
138.3 104.2 79.6 306.4
0.5 7.1 7.9 6.4
2 72 26
one year prior to failure, five cases were averaged, for failures in each of years 1992–1996. This gives an average of 38.1 for Model 1, for one year prior to failure (from the five values 50.6, 57.1, 14.8, 49.5, and 19.1 for scores in 1991 to 1995, respectively). Model 1 shows greater success (higher index) one year prior to failure in 1991, 1992 and 1994 than in 1993 and 1995. Table 3 also shows the standard deviation of the Failure Prediction Index for each year prior to failure. For five years prior to failure, only one year of data is used, resulting in zero for the standard deviation. Since Z-score data (Model 5) were available for only the years 1991–1996, comparisons are not made prior to 1991. Wheelock and Wilson (1995) consider banks in the US with an equity/asset ratio less than 2% to be a failure. The cut-off values for equity/assets scores from the Failure Prediction Index range from )2.0% to 1.7% for different years of failure, which are consistent with Wheelock and Wilson (1995). 4.3. Peer group comparison In this section a comparison is made between a weak Credit Union and its peer group as identified by DEA Model 1. The peer group represents the efficient Credit Unions on the DEA frontier that form a facet or part of a facet that the inefficient Credit Union is projected to, by the input oriented approach. This comparison permits identification of the deficiencies of the weak Credit Union with respect to its inputs and outputs in the model. The target inputs and outputs of the weak Credit Union are taken as a convex combination of the inputs and outputs of the peers.
hBCC
hB
0.92 1 1 1
0.59 1 1 1
The eighth failure in Fig. 4 is Credit Union L, which was identified by DICO as weak, and was purchased by Credit Union P. The purchase took place in 1995, and the comparison is made for the prior year 1994. Table 4 shows that the peers for Credit Union L (M, N, and O) are of a similar size, and that the equity/asset ratio of Credit Union L is much less than that of the peers. The k values total 100%, and indicate the percentage of each peer’s inputs and outputs that represent an efficient target for charter L. Of the three peers, the target for Credit Union L is most similar to Credit Union M ðk ¼ 72%Þ. The hBCC value of 0.92 for L indicates that all inputs for Credit Union L need to be reduced by 8% to reach the target. After this proportionate input reduction, further reductions are required if there is any remaining input slack, but this is not the case for Credit Union L. An increase in outputs (output slack) to reach the target is required (in this case, primarily an increase in equity, from $0.7 million to $9.8 million). The hB value indicates a slack adjusted efficiency measure of 0.59 for Credit Union L, compared to 1.0 for each peer.
5. Analysis Reasons for the variability in the Failure Prediction Index deserves further discussion. The methodology used in this research considers failures during any point in a calendar year to be failures in the same year. Thus, failures can be up to 11 months apart, yet they are considered as failures at the same point in time. The year of failure can be arbitrary if the date of failure is assigned to December rather than to January of the next year. The categorisation of a Credit Union as a failure by
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DICO to a specific date may not always be meaningful if a Credit Union has been faltering for some time. Performing the analysis on a quarterly basis would be more accurate. Credit Unions are considered a failure when DICO assigns a Loss Code, or considered to be the weak partner in a merger. Mergers ‘‘encouraged’’ by DICO in order to save a weak Credit Union, in all probability, have not all been identified and thus are not categorised as failures. Catastrophic failures such as a failure of an employer Credit Union due to a plant closure, and failure due to fraud, are not predictable with Models 1–6 (or perhaps by any other method). DICO management indicates that failures due to mismanagement are often believed to be due to fraud, but cannot be proven. The number of values of the Failure Prediction Index that is averaged in Table 3 varies from five values for one year prior to failure to one value for five years prior to failure. This small number of values makes it difficult to establish whether there is a statistically significant difference between the models, especially beyond one year prior to failure. The non-parametric Wilcoxon Rank Sum test indicates that for a two tailed test with a ¼ 0:10, we cannot reject the hypothesis that the distribution of the Failure Prediction Index is the same for Models 1, 2, 3, 5 or 6 for one year prior to failure. Only for Model 4 is there a statistically significant difference in the distribution with the other models, at a ¼ 0:05, for one year prior to failure. At three years prior to failure, we have only three data points to compare for the different models for the Failure Prediction Index; in this case, the only models that can be shown to be statistically significantly different are between Models 3 and 6, and between Models 4 and 6, at a ¼ 0:10. Thus, for Credit Unions with assets larger than $2 million dollars (1996 $) there does not appear to be a clearly best model for prediction of failure between the DEA Models 1, 2 and 3, the Z-score Model 5, and the equity/asset ratio in Model 6. DEA Model 4 does not fare well in comparison. Placing constraints on the hyperplane multipliers in Model 3 does not improve the results from Model 1; nor does projecting inefficient Credit Unions to the closest point on the frontier, as in
347
Model 2. For one year prior to failure, Model 1 has a higher mean Failure Prediction Index than the others, but also a higher variation. For two years prior to failure, Model 6 has the highest mean Failure Prediction Index and also a lower variation than Model 5, which has the next highest mean Failure Prediction Index. For three years prior to failure, Model 6 has the highest mean Failure Prediction Index followed closely by Model 1, with a similar variation. Overall, Model 6 with the simple ratio of equity/assets appears as good as any of the other models. However, while this measure is considered to be a universal indicator of financial health, it does not contribute any insights into the reasons for the problems, nor offers direction to management as to how to address them. Moreover, we must consider that the equity in a Credit Union comes from contributed capital by the members together with retained earnings. Since the very essence of these organisations is to provide the lowest cost lending and are essentially non-profit ones, not much capital is accumulated. Assets are, for the most part, loans to members or investments. It is assumed that loan losses are rare because the members’ deposits are at stake and the borrowers are closely tied to the community served by the Credit Union. All this adds up to unusually small equity/asset ratios even for healthy units when compared to other financial institutions. Therefore, Credit Unions can operate for years with a dangerously low equity/asset ratio because as long as loan defaults are low, they stay solvent. The importance of this background is that this makes the equity/asset ratio all important because a unit with a very low figure can fail relatively easily but it can avoid failure for years. Therefore, the ratio may appear to be a predicting failure far in advance of the actual problem but, in reality, it just states the fact that low equity/asset ratio can lead to problems. DEA Model 1 offers significant benefits to DICO as the regulator attempts to head off problems at suspect Credit Unions. By offering timely advice to Credit Union managers as to how to improve operations, ultimate failure may be prevented because managerially useful information can be provided to fix operating characteris-
348
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
tics other than the equity/asset ratio. Moreover, DEA provides useful information for inefficient Credit Unions that do not have low equity/asset ratios and management can make adjustments to avoid problems that may, in the long run, lead to a depletion of their equity. After all, equity erosion can be the result of operational losses as well as bad loans. Finally, if non-member targeted investments are made unwisely, the Credit Union may be in serious trouble if such investments should go bad. A vivid example of this can be seen in the US based Savings and Loan debacle of the past decade. Each of the models shows a higher mean Failure Prediction Index when DMUs analysed are restricted to those over $2 million dollars in asset size (1996 $), and slightly higher again when the restriction is placed at over $5 million dollars. Similarly, when Credit Unions that fail due to plant closure or fraud are not included, each of the models shows a higher mean Failure Prediction Index. The DEA models (1, 3, and 4) show higher values for the mean Failure Prediction Index when the slack adjusted hB measure is used than when the traditional hBCC measure is used (DEA Model 2 does not use the traditional hBCC measure).
6. Conclusions From the models included in this research, the equity/asset ratio and some DEA models appear to be equally competent in predicting the failure of Credit Unions. However, DEA Model 1 offers indicators of where the problems are and how to address them. Hence it should be the preferred tool for the regulator. Each of the models shows that failures, on an average, have lower scores than healthy units, for up to three years before failure, thus our hypothesis is proven. Prediction of failure is most reliable at one year prior to failure, and declines as we go further out. Prediction improves when only larger asset sized DMUs are included, and also when failures due to plant closure or fraud are excluded. Catastrophic failures due to the latter two causes cannot be predicted and should be excluded from all
analyses. DICO management believes that many cases of mismanagement are actually fraud but that cannot be proven. If this belief is true, then prediction of failure is more difficult than it would otherwise be. The models in this work do not consider the risk involved if a Credit Union has a large proportion of its assets in a single large loan or investment. Yet, this may be the most serious potential problem because a large loan default may well wipe out the entire equity of the Credit Union. Hence, size matters because the relative size between the firm’s equity and the largest loan or investment is a crucial survival issue. Improved results are expected if the models were run on a quarterly basis, rather than annually, as in this research. The availability of audited data would also be of benefit, although not likely to be available on a quarterly basis. The number of failures in any year as a percentage of the total number of Credit Unions is small. Prediction of failure of this small set of Credit Unions produces highly variable results for failures in specific years or certain years prior to failure. Attempting to apply a fixed cut-off score value to predicting failure could lead to high error rates in many instances. The equity/asset ratio provides as good a prediction of failure as any of the other methods, and validates the importance placed on this ratio by DICO. DICO’s Z-score does not appear to be a better tool than the equity/asset ratio. DEA Model 1 (with no multiplier constraints) provides results comparable to the equity/asset ratio when the slack adjusted hB measure is used, particularly for larger asset sizes. The use of the traditional input oriented measure ðhBCC Þ does not provide results as good as the results given by the slack adjusted hB measure. In this research, the use of multiplier constraints or projection to the closest point on the frontier, do not improve the DEA results. DEA provides prediction of failure comparable to or better than the DICO’s Z-score (particularly for larger Credit Unions) without the necessity of determining a set of arbitrary (or empirically derived) weights to improve the results. Moreover, DEA provides guidance to Credit Union management for improvements, the Z-score does not.
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
DEA Model 4, based on (but not identical to) the variables used by Barr and Siems (1991), Siems (1992), and Barr et al. (1993), does not perform as well as the other models analysed in this work. It should be noted that there are perhaps significant differences between the Model 4 used here and the original used by Barr et al. In this research, the VRS DEA model is used due to the wide range of asset sizes of Ontario Credit Unions. In the Barr et al. study, the CRS model was used, and the asset sizes were much more restricted. They did not use DEA results alone to predict the failure of US banks instead, DEA results were combined with the CAMEL rating to improve failure prediction. The benefits that DEA (such as Model 1) provides can be classified into three areas: • DEA does not require weights to be determined in combining ratios for ranking of Credit Unions. • DEA provides peer groups (relatively efficient Credit Unions) that inefficient units can compare themselves to, in order to improve their management practices. DEA also provides the amounts by which inputs should be reduced, and outputs might be increased, in order to become efficient. The regulator may use these results to monitor change in the units. • The combination of the DEA results in Model 1 with DICO’s Z-score or the equity/asset ratio can be used to identify those Credit Unions that might become a problem. A low score in any one model might not indicate a serious problem in the Credit Union under examination, however, a low score in any two or all of them, or a steady decline over successive quarters, should alert DICO to investigate further.
7. Concluding remarks This study forms a part of a continuing research program on productivity measurement technology in the Canadian financial services industry by the researchers at the Centre for Management for Technology and Entrepreneurship at the University of Toronto. Past studies include the examination of both retail and commercial branch
349
productivity in banks, Schaffnit et al. (1997); historical progress of the largest six banks over a 15 year period; Property and Casualty industry; and the Mutual Funds industry (papers are being prepared from all of these studies). During this particular work we focussed on the needs of the regulator. One of the most important outcomes is the verification of the results with the regulator’s own methodology, thus validating these findings.
Acknowledgements The authors would like to thank Kwabena A., Charles M., Bill F., Rudy B. and Francois G. for their invaluable support and their valuable time given so freely. This research was made possible by the generous financial support of the Toronto Dominion Bank, The Royal Bank of Canada, Bell Canada and the funding agencies of SSHRC and NSERC.
Appendix A. DICO Z-score A ranking of the performance of each Credit Union by DICO is performed in each quarter. A Z-score is calculated based on the following financial data, with the relative weight shown in brackets: Capital adequacy ¼ Regulatory capital /net assets (15) • assets are adjusted for imputed interest expense and taxes, • capital is reduced based on the likelihood of a sudden change in interest rates (shock test). Net interest income/average assets (7). Liquid assets/(deposits + borrowings) (1). Non-interest expense/average assets (6). Credit risk ¼ Loans delinquent over 30 days /risk weighted loans (8). From each ratio an individual Z-score is computed as (ratio ) mean)/standard deviation. The calculation for the standard deviation contains weighting based on the asset size of each individual Credit Union. Then the five Z-scores are weighted
350
P. Pille, J.C. Paradi / European Journal of Operational Research 139 (2002) 339–350
(the values in brackets above) and added together (the last two with negative sign). The variables Regulatory Capital and Equity are similar, and are used in different models. Regulatory Capital includes the effect of DICO’s ‘‘shock test’’, or effect of a sudden increase in interest rates. Also, Regulatory Capital is taken at the end of the fourth quarter and is not averaged over the year. Equity, as used in this research, is the average of the quarterly values, with adjustments for imputed interest expense (interest payments by a Credit Union are imputed in those cases where no interest payments, or very low values, are reported).
References Ali, A.I., Seiford, L.M., 1993. The mathematical programming approach to efficiency analysis. In: Fried, H.O., Lovell, K.C.A., Schmidt, S.S. (Eds.), The Measurement of Productive Efficiency. Oxford University Press, Oxford, NY. Ali, A.I., Lerme, C.S., Seiford, L.M., 1995. Components of efficiency evaluation in data envelopment analysis. European Journal of Operational Research 80, 462–473. Banker, R.D., 1984. Estimating most productive scale size using data envelopment analysis. European Journal of Operations Research 17 (1), 35–44. Barr, R.S., Siems, T.F., 1991. Predicting bank failure using DEA to quantify management quality. Research Paper, Federal Reserve Bank of Dallas. Barr, R.S., Seiford, L.M., Siems, T.F., 1993. An envelopmentanalysis approach to measuring the managerial efficiency of banks. Annals of Operations Research.
Bowlin, W.F., Charnes, A., Cooper, W.W., Sherman, H.D., 1985. Data envelopment analysis and regression approaches to efficiency estimation and evaluation. Annals of Operations Research 1 (2), 113–138. Charnes, A., Cooper, W.W., Wei, Q.L., Huang, Z.M., 1989. Cone ratio data envelopment analysis and multi-objective programming. International Journal of Systems Science 20 (7), 1099–1118. Charnes, A., Cooper, W.W., Huang, Z.M., Sun, D.B., 1990. Polyhedral cone-ratio DEA models with an illustrative application to large commercial banks. Journal of Econometrics 46, 73–91. Charnes, A., Cooper, W.W., Lewin, A.Y., Seiford, L.M., 1994. Data Envelopment Analysis: Theory, Methodology, and Application. Kluwer Academic Press, Dordrecht. Pille, P., 1998. Performance Analysis of Ontario Credit Unions, PhD dissertation, University of Toronto. Roll, Y., Golany, B., 1993. Alternate methods of treating factor weights in DEA. OMEGA International Journal of Management Science 21 (1), 99–109. Schaffnit, C., Rosen, D., Paradi, J.C., 1997. Best practice analysis of bank branches: An application of DEA in a large Canadian bank. European Journal of Operational Research 98 (1997), 269–289. Seiford, L.M., Thrall, R.M., 1990. Recent developments in DEA: The mathematical programming approach to frontier analysis. Journal of Econometrics 1–2 (46), 7–38. Siems, T.F., 1992. Quantifying management’s role in bank survival. Federal Reserve Bank of Dallas – Economic Review, January. Thomson, R.G., Langemeier, L.N., Lee, C.T., Thrall, R.M., 1990. The role of multiplier bounds in efficiency analysis with application to Kansas farming. Journal of Econometrics 46, 93–108. Wheelock, D.C., Wilson, P.W., 1995. Why do banks disappear? The determinants of U.S. bank failures and acquisitions. Federal Reserve Bank of St. Louis, Research Division Working Papers 95-013A.