The public accountability index: crafting a parametric disclosure index for annual reports

The British Accounting Review 36 (2004) 79–106 www.elsevier.com/locate/bar

The public accountability index: crafting a parametric disclosure index for annual reports David Coya, Keith Dixonb,* a

b

University of Waikato Management School, Hamilton, New Zealand Financial Management Center, Open University Business School, Michael Young Building, Walton Hall, Milton Keynes MK7 6AA, UK Received 22 May 2002; revised 19 June 2003; accepted 3 October 2003

Abstract The methodology used to craft a disclosure index with parametric statistical properties is described and explained. The index is developed from a public accountability perspective using stakeholder opinions captured via a delphi exercise. Named the Public Accountability Index (PAI), the index is applied to the annual reports of the New Zealand universities, 1985 –2000. This was a period of radical change for the public sector, and the index scores reflect the wholesale changes in reporting over the period. PAI is offered as a generic stakeholder approach for the development of disclosure indices to measure annual reports of groups of organisations in any of the private, public or so-called third sectors. q 2003 Elsevier Ltd. All rights reserved. Keywords: Public accountability index; Delphi exercise; Stakeholder

1. Introduction Disclosure indices are an oft applied method in accounting research, particularly in studies of annual reports, being used to provide a single-figure summary indicator either of the entire contents of reports of comparable organisations or of particular aspects of interest covered by such reports (e.g. voluntary disclosures and environmental disclosures) (Ahmed and Courtis, 1999; Coy et al., 1993; Marston and Shrives, 1991; Singleton and Globerman, 2002). Here we report on the Public Accountability Index (PAI) and its use in a study of annual reports of universities located in New Zealand. The novelty of the PAI is * Corresponding author. Tel.: þ 44-1908-659168; fax: þ 44-1908-655898. E-mail address: [email protected] (K. Dixon). 0890-8389/$ - see front matter q 2003 Elsevier Ltd. All rights reserved. doi:10.1016/j.bar.2003.10.003

80

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

that the numbers generated by it have numeric properties consistent with use in parametric statistical analyses. A further source of originality is that our study is longitudinal, not only in the obvious sense of analysing a series of annual reports spanning more than a few years (e.g. see Cameron and Guthrie (1993)) but also in two further and inter-related senses. First, we have studied these reports continuously since 1990, measured them and analysed their waxing and waning. Second, throughout this study period, we have been engaged in a process of index crafting. The PAI has emerged from indices of lesser utility, previously published in relation to 1985 –1989 reports (Dixon et al., 1991),1 1985 –1990 reports (Coy et al., 1993)2 and 1990– 1992 reports (Coy et al., 1994).3 It was first used in 1997 to score the universities’ annual reports for 1996. We employed it subsequently, including to 2000 reports and, retrospectively, to reports for 1985, 1988 and 1992, thus giving results reported here covering reports spanning sixteen years. Going back to our first, a significant point is that even if in 1990 we had had the knowledge and skills that we have now in disclosure-index construction and application, we could not have constructed the PAI then because of its contingent nature vis-a`-vis developments in reporting over the past decade and a half. We shall address the important issue this raises about whether an index developed in one reporting era is applicable to a previous era. Initially, our motivation for the study was as academic employees with professional accounting backgrounds. We were concerned about educational and staff-related aspects of university resource attraction and allocation, and in pursuing this concern found that little relevant information was available anywhere from the universities’ authorities, let alone reported publicly either in their annual reports or otherwise (Coy and Pratt, 1998; Tower and Coy, 1990). Our interest coincided with moves among accounting and governmental bodies to increase accountability of public sector organisations, through improved reporting (McCulloch and Ball, 1992; New Zealand Society of Accountants (NZSA), 1987; Public Finance Act, 1989). With notable exceptions, the universities were slow to respond to these moves but in the early 1990s they were ensnared in them, 1

This was a dichotomous index, being one in which each component item is given a score of ‘1’ if the item appears in a report, regardless of how minimal or comprehensive its disclosure, and a score of ‘0’ otherwise. Although sufficient initially, this instrument was somewhat crude. It did not detect trends that were evident merely from examining the reports: that, in each year, the comprehensiveness and understandability of disclosures in reports of some universities were better than in others; and that, from year to year, disclosures in each university’s reports were changing. 2 This was the Accountability Disclosure Score (or AD-score) index. Each of 43 disclosure items was evaluated for extent of disclosure using a four-point ordinal scoring system of up to ‘3 ¼ excellent’, giving rise to the index being described as polychotomous. Furthermore, each item was weighted according to perceived importance on a three-tier scale, up to ‘3 ¼ high’. Reports for six consecutive years of all 39 tertiary education institutions in New Zealand were ranked and this showed, among other things, that their contents improved in 1990 compared to previous years. 3 This was Modified Accountability Disclosure Score (or MAD-score) index. Arising from the continuing enhancement of report disclosures in the early 1990s, we modified the AD-score. The modified index had only 25 items but these were larger in scope than those in the AD-score and the ordinal scoring scale was extended from 4 to 6 points. Using this index showed that for reports from 1990 to 1996, later reports continued to out-rank earlier reports in total and across all categories of information (see also Coy and Dixon (1996)).

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

81

as well as in other financial and managerial changes (e.g. formula funding and imposing their own higher fees, rational approaches to resource allocation) (Coy et al., 1991; Education Act, 1989; Ministry of Education, 1991). From these initial influences, our study has proceeded primarily within a public accountability perspective (Coy et al., 2001), which is rehearsed in Section 2. The rest of the paper has two main themes: first, disclosure indices in general and the PAI in particular; and second, trends in university reporting since 1985 indicated by the PAI numbers. Sections 3 and 4 mostly deal with the first of these, and Section 5 the second, although there is some crossover. We conclude by assessing the suitability of the PAI construction method as a generic approach for the construction of disclosure indices.

2. Public accountability The broad objective for the PAI is to measure the quality of university annual reports from a public accountability perspective. This theoretical stance suggests a broad focus on reporting rather than one that may be concerned with narrower issues, such as assessing financial viability only or educational performance alone. In the context of contemporary New Zealand universities, public accountability refers to the reporting of comprehensive information about the condition, performance, activities and progress to all those with social, economic and political interests (Coy et al., 2001). Three reporting paradigms can be discerned, namely, stewardship, decision usefulness and public accountability. The stewardship paradigm developed from the traditional master– servant relationship and entails accountability between agent and principal that is often conducted in secret. Decision usefulness has been the ascendant reporting paradigm in the USA and other English-speaking countries since the 1970s. It is motivated by neoclassical economic considerations and the information needs of existing and potential investors, lenders and similar suppliers of capital. By contrast, the public accountability paradigm takes a broader perspective that encompasses social, political and wider economic interest in reporting organisations among numerous parties, including subsuming the narrow economic interest of the decision usefulness paradigm (Coy et al., 2001). The ideas of Scott (1941), Normanton (1971) and Chen (1975) are used by Coy et al. (2001) to argue for a form of public accountability that requires open disclosure to all citizens who have an opportunity to make criticism. Public accountability seeks justice, truth and fairness in reporting, as argued by Scott; and acknowledges a primary stewardship responsibility to society and a secondary one to stockholders, as claimed by Chen. Normanton points out that [financial] accounts may conceal more than they reveal and, therefore, that accountability requires disclosure of reasons for, and explanations of, actions taken. Furthermore, accountability reports are prepared by accountors who are concerned to justify themselves and may avoid issues that have caused them embarrassment, although it is precisely these issues that should be the concern of public accountability. Thus, the idea of public accountability and its application are not as

82

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

clear-cut as might be presumed, particularly in the light of recent changes in public perceptions of annual reporting. Regarding the status of the public accountability paradigm, extending coverage and producing more copies of published annual reports has had a major impact on the nature of annual reporting in New Zealand and in its public sector. Since the reforms of the 1980s public sector reports have been produced with public accountability as an important espoused objective of reporting (e.g. see NZSA (1987) and Institute of Chartered Accountants of New Zealand (2002)). Reports now extend beyond reporting merely the financial flows to include more comprehensive information about such matters as objectives, achievements and service performance.

3. Disclosure indices An index comprises numbers that encapsulate, in single figures, objects in the set that one wants to measure and that are capable of measurement. Each number in a valid and reliable index is reached uniformly by determining scores for each of, possibly, many component items, which have been identified as relevant to the set and the purposes of the index. These component-item scores are then combined in a meaningful way in order to ascertain the index number for each object in the set. Retail price indices are everyday examples of the concept: they express the levels of retail prices in a particular location at various times through index numbers computed from a weighted average of prices of commodities bought by consumers. As Marston and Shrives (1991) point out, three scales of measurement, namely ordinal, interval and ratio scales, are generally recognised as within the capacity of indices. But constructing a disclosure index with interval scale properties, let alone ratio scale properties,4 appears to have been shied away from for being problematical (see reviews by Botosan (1997) and Singleton and Globerman (2002)). However, our particular interest in the possibility of an index with ratio power was sparked by Siegel (1993). He argues that social science researchers often use ordinal data when these might easily be collected as interval or ratio data by slight variation in research technique. Ingram and Robbins (1992) attempted this by applying a magnitude scaling technique to items measured ordinally in a study of the information needs of users of USA governmental reports by Jones et al. (1985), and concluded that a valid ratio measure could be achieved. Two fundamental differences exist between ratio scale indices, as exemplified in a price index, and the disclosure indices to be found in accounting research. First, price indices are open ended, whereas disclosure indices are usually constructed in such a way as to have a maximum score, no matter that reports in a continuing series might reach ever-higher levels of quality. Second, although a price index measures a wide variety of items, they are all measured in terms of the same externally determined value (i.e. market prices expressed in local currency). In contrast, items comprising a disclosure index that is not of the dichotomous sort are measured by a finite range of values assigned to descriptive 4 Interval and ratio scales both measure the distance between items on their scales, but an interval scale starts from an arbitrary point whereas a ratio scale starts from an absolute point, zero.

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

83

criteria established internally by index constructors in an effort to reflect varying levels of quality in how each index item appears across the reports they are studying. But all that seems to be achieved by this method is that disclosures of particular items are deemed equivalent to, better than or worse than one another, giving rise to an order or ranking of individual items, whole categories of items and entire reports. Put another way, it is problematical to use differences in disclosure index scores for two reports (e.g. 60 and 66) arrived at using this method as denoting an interval on a scale. One should not, for example, interpret the 10% difference between index scores of 60 and 66 as indicating that one report is 10% different from the other; or infer interpretations that involve proportions or express differences in ratio terms (e.g. this balance sheet is 10% better than that balance sheet). On the other hand, a 10% difference between numbers in a price index can be interpreted as a 10% difference in weighted average prices ruling at the times to which the numbers relate. The significance of disclosure indices having only ordinal properties is that the statistical tests involving the index numbers should be limited to non-parametric types, such as the Chi-Square test. Theoretically, the calculation of even a simple measure such as the arithmetic mean of a series of ordinal index numbers is invalid. Nevertheless, it is easy to fall into the trap of treating ordinal scale index numbers as having interval and ratio scale properties, and using them in parametric statistical analyses, which can be a recipe for making errors and misinterpreting data. Among the contemporary issues that Marston and Shrives (1991) raised was the propriety of disclosure indices that departed, first, from dichotomous scoring of the index items and, second, from giving each item in the index the same weighting in computing the index numbers. Having developed and used indices that departed on both counts,2,3 we have grappled with these issues, which remain topical. This is exemplified by Ahmed and Courtis (1999), who claim that unweighted, dichotomous disclosure indices reduce subjectivity and are the norm. However, we suggest that the latter claim is an overstatement and that the former is misguided. Aside from taking a view that subjectivity, political process and ethnocentrism are endemic to all research (Davis et al., 1982; Pettigrew, 1985; Silverman, 1985; Silverman and Gubrium, 1989; Turner, 1989), thus making the objective – subjective dichotomy somewhat questionable, it is our experience that subjectivity is endemic in studies using disclosure indices. It derives from the social constructions that disclosure indices are employed to measure (e.g. environmental information, information that is decision-useful, information facilitating public accountability). It also derives from the theoretical stances adopted explicitly or implicitly by researchers about who should disclose what to whom. Thus, not only are indices instilled with subjectivity through choices about which items to include in an index and which to omit, no matter who makes these choices and how they make them (e.g. by the researchers reviewing the literature or consulting focus groups), but also how to include them. We can give two reasons to support our claim that dichotomously scored, unweighted indices suffer from problems with subjectivity. First, treating all report disclosures as being of equal rank and weight, even though there are obvious differences in content and importance, is itself a subjective decision. For example, a ‘statement of objectives’ is one of the items in the MAD-score index (Coy et al., 1994). Reporting practice showed that its

84

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

disclosure could range from a brief statement with little elaboration to one that was, in the authors’ opinion at least, comprehensive and informative, encompassing such features as a mission statement, goals and performance targets expressed, quantitatively and qualitatively, in specific, concise, understandable and practical terms within a given time frame. By not discriminating between such variations in quality, index constructors would be awarding the same score to disclosures that are not the same. This weakness could be addressed by replacing the ‘statement of objectives’ and similar broad items with the many smaller items of which they are expected to be comprised. However, not only would that make an index unwieldy, it would also be less flexible. For example, useful information disclosed as part of a ‘statement of objectives’ that was not included among the subitems list would not be recognised by the index. Second, all index items are accorded equal importance regardless of whether their absence (or presence) is fundamental to the overall quality of a report. For example, in the context of university reporting, the general opinion seems to be that disclosures about prime activities, such as teaching and research, are more important than disclosures about subsidiary activities, such as student accommodation and administration. The effect of not weighting items according to their perceived importance is to choose to give the same weighting to fundamental matters as to incidental matters. Moreover, weightings also allow greater recognition of items that are inherently extensive (e.g. a balance sheet) compared to items that are inherently limited in extent (e.g. a statement of movements in equity), although breaking the balance sheet out into several component items of the index would have the same effect. Indeed, supposedly unweighted disclosure indices are weighted according to how many items their constructors include from each broad category of disclosure. For example, Botosan (1997) indicates that her DSCORE has five categories, each with a different number of component items, so even if its weightings are removed and each item was to count equally, it would still be weighted (Singleton and Globerman, 2002).5 We believe that so-called subjectivity is something one has to both live with and take care to ameliorate. Limiting oneself to unweighted, dichotomous indices not only prevents measurement of disclosures rising above the mundane but also could give rise to a false sense of avoiding subjectivity. Furthermore, these limits mean that disclosure indices are never going to be more than ordinal scales of measurement, with the attendant concerns raised earlier about their limitations in statistical analysis. Another issue is whether to include in an index, disclosures that are mandated by statute. One argument for omitting them is that as they are mandated they will be in all reports, and so will not cause differences in scores attained by reports, provided that reporting organisations do not flout the law. However, our experience is that quality differences do arise even among mandatory disclosures, not only between organisations but also in reports of the same organisation from year to year. 5 In practice, the use of weightings often does not materially affect index results (Spero, 1979; Cooke, 1989). This finding is often used as a reason for not using weightings, but Botosan (1997) took the opposite view: that is, that using weightings was intuitively more appealing and, because the unweighted index would not have led to different conclusions, the results could not be called into question purely on the issue of using weightings.

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

85

4. Constructing the PAI On reflection, the way we constructed the PAI comprises a six-step model, as follows: 1. 2. 3. 4. 5. 6.

Identifying the objectives of reporting for the sector of interest Reviewing contemporary reporting in the sector of interest Determining the objectives of the index Identifying appropriate disclosure items and report qualitative characteristics Obtaining appropriate stakeholder validation of index items Crafting and testing the index.

Explanations of the first three steps are subsumed in earlier sections, and so we limit this section to explanation and discussion of the last three steps, as per the sections below. While this model may have wider application, anyone using it is cautioned that disclosure indices are ephemeral in nature and can measure annual reports only in terms of the standards and expectations of society when they are constructed. As changes occur in society’s expectations and the issues of concern to various of its members, so there are arguments for modifying a disclosure index purporting to be measuring reports from a public accountability perspective. Not only does this mean that the specifics of the PAI are applicable only to New Zealand universities, and for a limited time, but it also begs the question of whether generic principles of index construction can be specified, and so whether this model is transferable. 4.1. Step 4: identifying appropriate disclosure items and report qualitative characteristics The method that an index constructor can use to identify items and qualitative characteristics depends on the state of annual reporting by the sector of interest. Where reporting is well established and the report audience is mature, well defined and knowledgeable, then report items may be identified by seeking suggestions and opinions from members of that audience. This approach has the advantage that report users’ opinions may be captured without the risk of influencing those opinions by provision of a prompt list or similar, such as used in the questionnaire survey undertaken by Coy et al. (1997). At the other extreme, where there is no knowledgeable report audience, persons constructing an index may have little choice but to draw on literature sources for potential items, or even only for theories from which to deduce potential items. When we were carrying out the PAI construction in 1994 and 1995, the New Zealand university sector was inbetween these positions. Annual reports were only emerging as a significant means of communicating with stakeholders and the audience was on a learning curve. We met this contingency by basing selection of component items for the PAI on a review of extant practice and a literature search and review, in which 15 sources were identified as useful to performing this step. Nine sources addressed university annual reporting (Committee of Vice-Chancellors and Principals, 1994; Coy et al., 1993, 1994; Engstrom, 1988; Fischer and Gordon, 1991; Gray and Haslam, 1990; Ministry of Education, 1991; NZSA, 1993; Public Finance Act, 1989) and the others related to educational, managerial and general performance indicators and statistics (Cave et al.,

86

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

1991; Cutt et al., 1993; Department of Employment, Education and Training (DEET), 1994; Gannicott, 1994; Johnes and Taylor, 1990; Performance Indicators Task Force, 1989). From these procedures, most of the 130 disclosures and report qualitative characteristics listed in Appendix A were identified for initial consideration as component items in the PAI. 4.2. Step 5: obtaining appropriate stakeholder validation of index items The main shortcoming of the information obtained thus far was that it took us little further than the items and rationale of the MAD-score index. We wanted the index to capture, as far as possible, the essence of public opinion about what should be disclosed and the relative importance of the items. We decided to carry out a delphi opinion-seeking exercise with university stakeholders, in preference to such alternatives as ordinary questionnaires, focus groups and interviews. The delphi exercise also served to test the relevance and importance of possible index items. Delphi is a means of soliciting opinions, usually on complex matters, from interested or expert participants, whose identities are unknown to each other and who work independently (Linstone and Turoff, 1975). The method comprises rounds in each of which participants are asked for their opinions on the subject of interest and then these opinions are collated and summarised, the summaries being used in the following round as part of asking for further opinions. Each round allows, among other things, participants to reconsider their position in the light of the opinions of other participants. This opportunity for iteration and comparison is thought to lead to more considered and reliable responses than might be the case with ‘one time only’ data collection methods. The number of rounds staged depends on how many it takes to reach some stability in the responses, and whether the participation of panellists can be sustained. Delphi is a practicable, cost-effective method of capturing the opinions of participants, whilst avoiding problems of peer pressure, undue influence and other contamination, which can occur when people meet in groups and similar situations. The delphi exercise involved a theoretical sample of 39 participants selected purposefully (see Patton (1990) on qualitative sampling) to comprise the five panels of university stakeholders shown in Table 1. The panels were identified in Coy et al. (1997) Table 1 Details of delphi panels Panel name

Panel description

Size

Internal citizens Academic and general staff who served on university councils 7 External citizens University councillors not employed by universities 9 Student representatives Student representatives on university councils 6 Oversight agents Five auditors, two MPs and two Education Ministry officials 9 Report preparers Four finance and four information specialists from six universities 8 The 22 participants who were university council members were drawn from all seven of the universities existing in 1994.

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

87

which reported that, contrary to established theory about external financial reporting, the audience for tertiary education annual reports in 1993 was more numerous among parties closely associated with these organisations (e.g. scholars, administrators and governors) than among resource suppliers, oversight bodies, analysts and the public. Further investigation in 2001 indicates that, proportionally, the closely associated audience was even greater than in 1993 (Dixon et al., 2002). In the first round, participants were sent a questionnaire featuring most of the items shown in Appendix A. They were asked to indicate the importance they attached to each one featuring in annual reports using a four-point scale (0 ¼ should not be disclosed to 3 ¼ extremely important to be disclosed), as well as to add any further items they thought worth considering. The first round results were summarised and returned to participants with an invitation to reconsider their initial responses, including considering a few additional items proposed by participants, making the total items up to 130. After only two rounds, it was considered that enough consensuses had been reached within all panels;6 indeed, the level of agreement among panels was also high. 4.3. Step 6: crafting and testing the index We transformed the delphi panels’ opinions into a draft index in a quite complicated process reminiscent of crafting. That is, various activities proceeded in concert, rather than serially, and the process was characterised by trial and error (see Mintzberg (1987), regarding this metaphor). The subsections below elaborate on six parts of the crafting process. In essence, we drafted and redrafted the index, testing it every so often on sample annual reports. Then, it was exposed to external comment and more modifications were made. The final index comprises the 58 items arranged into eight specific categories as shown in Appendix B. 4.3.1. Determining items to be included The 58 items included in the PAI were distilled from the 130 tested in the delphi exercise. Some pragmatic matters had to be resolved, mostly concerning overlap and repetition between items, and ways to reduce or avert the need for so many items as would make the index unwieldy. For example, in the delphi study it was established that the ‘balance sheet’ and ‘changes in financial equity’ are, respectively, ‘extremely important’ and ‘important’ disclosures. As ‘changes in financial equity’ are part of the disclosures within the ‘balance sheet’, it was deemed more appropriate to capture the ‘changes in financial equity’ as part of the ‘financial position statement’ item rather than treat these as separate index items. Conversely, there were similar situations where it was deemed more appropriate to keep items separate. For example, the statement ‘cost of services’ might be regarded as a part of the ‘financial performance statement’, but because of their diverse foci (i.e. the financial performance statement is concerned with the whole organisation, the cost of services statement with the allocation of resources to the different segments of 6

The standard deviation for every category of disclosure item for every panel, and the average standard deviation for all items for all panels was lower in round two than in round one, indicating greater uniformity of opinion.

88

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

the organisation, which is an important issue in universities), it was decided to treat these as separate items in the index. Another important pragmatic matter was how to deal with what accounting standard setters often refer to as the qualitative characteristics of reports (e.g. understandability, reliability, verifiability, comparability and consistency). Although these are important among report recipients (see Coy et al. (1997) and Dixon et al. (2002)), most were excluded from the delphi exercise, either because they seemed subsumed by using polychotomous scales to measure disclosure items or because practical measurement methods to assess them in individual reports are not available. However, three qualitative characteristics were tested in the delphi, namely, timeliness, accessibility and relevance. Although all three featured prominently in panels’ ratings (see Appendix A), in the end we decided that from a practical standpoint only timeliness could be included in the PAI, because it is a distinct and separate quality that could be measured in a practical sense by recording the time at which reports are issued to the public.7 4.3.2. Determining the categories used to classify items We have found it helpful in previous indices to structure an index into categories into which index items are classified. This aids both moving from item scores to the total index number for a report, and analysing how different parts of reports compare from one university to another and change over time. For the PAI, we identified three broad categories in the theory development, namely, Overview, Financial and Service disclosures, and subdivided them to make eight specific categories. Overview items were subdivided into items about the report and about the university; and Service items were subdivided into general, teaching, research and community service items, with Service: teaching items being further subdivided into input, process and output/outcome items. These specific categories seem to us to have theoretical underpinnings and to be practical. However, some items overlapped divisions and caused another problem to surmount. For example, delphi panels supported disclosure about departmental reviews that in practice cover both teaching and research, but we decided to classify the item only within the index under ‘service: teaching’ and not to repeat it under ‘service: research’. 4.3.3. Devising evaluation criteria Earlier we discussed the issue of polychotomous versus dichotomous methods of scoring index items. For PAI, we adopted a polychotomous approach on the grounds that all annual report contents may be disclosed with variable amounts of detail and understandability. Thus, for each item we devised criteria that would facilitate scoring each item in the index as it appeared in the reports we would be analysing. However, 7 We decided that to have included accessibility and relevance in the delphi was an error. The first comes within qualitative characteristics that are impractical for independent assessors to measure without undue effort and cost (although nowadays internet availability might be an easily collected proxy measure). The second comes within qualitative characteristics captured by inducing and confirming the individual index items using data from stakeholders.

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

89

in order to try to confer interval numeric properties on PAI numbers, we made a significant departure from our previous attempts at index construction. In the MADscore index, items were scored using a six-point scoring scale, each point with a criterion, including zero for no disclosure of the item at all. In PAI, using the approach recommended by Siegel (1993) and adopted by Ingram and Robbins (1992), we set only one ‘standard’ criterion or benchmark for each item, equivalent to a score of ‘5’. Each of these benchmark criteria were devised in keeping with the public accountability objectives of the index and the best examples of reporting practice among reports we had analysed in the study up to then. Thus, we applied our observations of contemporary university annual reports, along with stakeholder opinions gathered in the delphi exercise and the criteria used by Coy et al. (1994) in the MAD-score, which we decided were robust and should be closely followed.8 Examples of criteria are shown in Appendix C.9 The benchmark criterion score of 5 is not only given to disclosures fulfilling said criterion but also it is used to judge what score to give to disclosures that differ from the benchmark. This judgement is important as it not only entails assessing a disclosure as being worse than the benchmark but also by how much, thus attracting scores from 0 to 4; or conversely, as better than the benchmark but also by how much, thus attracting scores from 6 to 1. For example, a disclosure judged 10 – 29% better than the benchmark would attract a score of ‘6’; and a disclosure judged as being better by between 30 and 49% would score ‘7’. Admittedly, to state precisely what represents better or worse by 10 – 29% or 30 –49% is impracticable, but given broad guidelines, it seems not unreasonable for evaluators to arrive at some consistency in scoring. Initially, we tried using a score of ‘10’ as equivalent of the benchmark criteria, but found that this scale (i.e. 11 points) was longer than necessary and made it difficult to be consistent across similar disclosures in different reports. 4.3.4. Allocating weightings To be an index, PAI has to provide a single number that aggregates scores of all items, although the individual item scores are also useful to measure and analyse differences between reports and particular disclosures. We continued with the logic of weightings, as per our previous indices, applying them to combine the scores of the eight specific categories. We adopted ‘100’ as the total weightings available and spread them among the categories, giving the joint highest weighting (25) to Financial items and Service: teaching items, and the lowest weighting (2) to Service: community service items. This allocation was arrived at by using both the empirical evidence provided by the stakeholder delphi panellists (see ‘importance’ scores in Appendix A) and an iterative process requiring us to exercise our judgement that fairly reflected panellists’ opinions. At the same time, weightings totalling 100 were allocated to the items within 8 Despite shortcomings in the MAD-score index for our purposes, anecdotal evidence indicated that report preparers regarded it as a fair and reasonable way to evaluate their handiwork. Some preparers were trying to improve their reports using as benchmarks its criteria, alongside examples from sister institutions’ reports that were evaluated as ‘excellent’ under these criteria. 9 The criteria for all 58 PAI items are available from the first author and at the website: www.mngt.waikato.ac. nz/tears

90

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

each category. Where two or more items were combined into one index item, we exercised our judgement about the weighting for the composite item. All weightings are shown in Appendix B. Subsequently, we performed sensitivity tests on applications of the index to the reports for 2000 (see further details below) and found that there was little impact on report rankings through using different weighting configurations. Rather than interpret this as a reason for axing weightings, we preferred to follow Botosan (1997) and retain them on the grounds that they put into practice what we believe intuitively, that not all disclosures are of equal importance, and that they do not add controversy to the index because they have little effect on the results. 4.3.5. External validation The draft PAI was sent for comment to an experienced researcher of public sector reporting and an experienced university report preparer. Both sets of comments supported the general framework of the index, insofar as the interval scoring system and the adopted categories are concerned, these being important innovations of the PAI. The academic in particular provided many detailed comments reflecting the subjective nature of this exercise to capture and use stakeholders’ opinions. After considering all the points alongside the opinions collected in the delphi exercise (of which neither commentator was aware), we left most items alone. For example, the academic suggested a more extensive section on capital assets and long-term financing, which would subsume the item ‘total value of estates’; greater recognition for prospective information, including prospective revenues and costs for up to five years ahead; and ‘overhead allocation’ to be subsumed within ‘cost of services’. An example where we made changes following the comments received was to provide more detail about tuition fees, for which we modified the criteria for the ‘financial performance statement’. The academic felt that the role of universities as critic and conscience of society was not adequately catered for in the index, and as a result of this suggestion, the criteria for ‘national community service’ was modified to make specific reference to this aspect of university services. The academic questioned two matters about which we empathised but have done nothing, namely: whether the inclusion of items ‘employer satisfaction’ and ‘pass rates’ implies a narrow view of the raison d’eˆtre of university education, and can lead to inappropriate behavioural reactions within the universities; and whether it is appropriate to categorise students under the heading ‘inputs’. The academic also suggested that the presented order of categories listed in the index should be changed such that the Financial category appeared last, reflecting the de-emphasis given to this kind of information in a public accountability perspective of annual reports. The point was acknowledged and it must be stressed that the order of categories presented in the index is not intended to imply priority or importance: the weightings signal these matters. The preparer thought that the index was comprehensive and could not detect any obvious weaknesses or omissions. But the preparer thought that the timeliness benchmark was too demanding, bearing in mind the difficulty of getting timely

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

91

co-operation from both auditors and academics in publishing reports.10 Overall, the preparer considered the weightings reasonable, although open to further refinement. The preparer wondered whether university reporting was now providing too much information, rather than too little, thus clouding the understandability of reports.11 PAI addresses this matter partially; excessive disclosures that are difficult to comprehend are penalised with a lower score. 4.3.6. Using PAI to score reports Timeliness was scored using data collected every year after 1990 by noting the date on which reports were received by post, following prompt requests to each university.12 For the other items in the PAI, firstly, disclosures were scored independently by both authors, in 199713 and 2001. Each author read the whole of a report and allocated every part (e.g. phrase, figure, sentence, statement, graph, picture) to one or more of the disclosure items in the index, noting the page number(s) for further reference. Many pages are narrative and each subsection and paragraph had to be read closely. This tricky process was timeconsuming, absorbing as much as two hours for a report that was comprehensive and lengthy.14 Then, each author visited the place(s) in the report that qualified for points under each index item in turn, compared the aggregate of these with the criteria and scored the item. Secondly, the two authors collaborated to check, scrutinise and compare between universities’ reports, and, in the case of 2000 reports, compare them with reports for 1996. One author checked the score sheets to ensure that both authors had identified the same pages in the report as relating to each disclosure item. Differences occurred in about 15% of total page references and the author visited the page(s) to ascertain whether pages missed by one author or the other should be included. Then, the author went through all the scores for all the reports, whether or not the independent scores agreed or not, to review every score. For 1996 reports, matrix-shaped score sheets were created for each item using any subitems in the benchmark criterion, and these were used to reconsider and, where necessary, revise scores, taking into account how one report compared with another. For 2000 reports, the item-by-item 10 On average, delphi panels opined that reports should be published within 16 weeks of balance date. Panel averages were: Internal citizens 21 weeks, Preparers 20 weeks, External citizens 16 weeks, Oversight agents 16 weeks and Students 8 weeks. To devise the PAI benchmark criterion of between 12 weeks and 13 weeks and six days, we relied on the opinions of the non-employee stakeholders (i.e. external citizens, oversight agents and students) because these stakeholders are less likely to have continuing access to alternative sources of annualreport-type information. We used 14-day intervals to give higher or lower points, and so reports published more than 21 weeks after balance date score a ‘0’, which is higher than any panel means listed above. 11 This signals the possibility of a further qualitative characteristic, namely ‘information excess or overload’, being the provision of so much information about a particular topic that it becomes difficult for stakeholders to understand its meaning. 12 For earlier years, the date of the auditors’ reports was used to estimate publication dates. 13 For 1996 reports, the exercise was also performed for the other 31 tertiary education institutions in New Zealand. 14 The average number of pages in the 2000 reports was 87 with a range from 72 (Victoria) to 110 (Otago). Similar lengths applied in 1996, lengths having built up since 1985, when reports were between 10 and 15 pages.

92

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

comparison of reports and scores included referring to the 1996 matrix score sheets and notes. Thirdly, the detailed PAI scores and page number analysis for 1996 and 2000 reports were sent to a preparer at each university for comment. Although we had been prepared to revise scores as appropriate, we received a few responses to our inquiries and these were mainly supportive, and so no changes were made. Subsequently, PAI scores were prepared for the reports for 1985, 1988 and 1992, relating to which we had experience of applying forerunners of PAI. This preparation included comparing reports in these years with reports for 1996 in particular and with each other.

5. Applying the PAI 5.1. How reports changed over the 1985 – 2000 period The summary PAI scores at regular intervals from 1985 to 2000 for the eight institutions that are now universities are shown in Table 2. The index numbers for 1985 and most of those for 1988 bear out that reports in the mid-1980s were brief documents published between 6 months and over 2 years after balance date. They were comprised almost entirely of unconsolidated, modified-accruals-based financial statements reflecting the keeping of accounts for general and specific funds. Save in financial statements, they reported little about the universities’ prime activities, teaching and research, either quantitatively or qualitatively; and in financial statements the value of resources attracted and consumed on these activities were highly aggregated. Stakeholders could learn little

Table 2 Summary PAI scores: NZ University annual reports, selected years 1985– 2000 Institution

Rankings (min PAI no. ¼ 20)

PAI scores 1985

1988

1992

1996

2000

University of Otago University of Auckland Auckland University of Technology University of Canterbury Massey University Victoria University of Wellington University of Waikato Lincoln University

11 11 n.a. 11 11 10 8 10

12 13 12 11 20 12 40 11

42 35 41 42 49 54 58 44

70 54 70 58 66 84 72 67

83 71 68 65 64 61 58 54

Average Maximum Minimum Range

10 11 8 3

16 40 11 29

46 58 35 33

68 84 54 30

66 83 54 29

n.a.: not available.

1988

2 1

1992

1996

2000

5 8 7 5 3 2 1 4

3 8 3 7 6 1 2 5

1 2 3 4 5 6 7 8

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

93

that was germane, such as how resources were allocated among students, courses, projects and disciplines (Table 2).15 The index numbers for later years indicate that in the decade or so since there has been radical change in the coverage of the annual reports, although the pattern of change has not been uniform. One university, Waikato, was proactive in extending its reporting, as reflected in its score of 40 in 1988, which was double that of the next university report, and it still had the highest score (of 58) in 1992. But by 1992, all the universities were making marked improvements in their annual reports, and this continued between 1992 and 1996, resulting in the average index number increasing by 22 points (48%) from 46 to 68. However, this appears to have been the limit of the collective improvement because between 1996 and 2000 the scores of three of the universities (i.e. the Universities of Auckland, Canterbury and Otago) increased appreciably, two (AUT and Massey) barely changed and three (i.e. Lincoln, Victoria and Waikato) fell appreciably, leaving the average PAI number in 2000 slightly down at 66 from 68 (3%). Moreover, these changes mean that rankings for 1996 and 2000 are almost inverted. Victoria and Waikato are ranked first and second in 1996, and sixth and seventh in 2000. Otago and University of Auckland are ranked first and second in 2000 compared with third and eighth, respectively, in 1996.16 As to explaining why changes have occurred between 1996 and 2000, our first inclination is review the substantial changes that have occurred among senior personnel in the eight universities and to obtain their opinions on accountability and the roles of annual reports in that process. This is being investigated, along with an analysis of trends in item scores. We have used Appendix B to give a foretaste of this, by including the detailed item scores and category scores for two universities for 1996 and 2000. One university, Auckland, shows the biggest increase in PAI number between the two years; and the other, Victoria, the biggest decrease. The items’ scores of these universities reflect some of the overall changes among the eight categories of the PAI between 1996 and 2000. On average, improvements occurred in timeliness17 and ‘service: general’ disclosures, and declines occurred in the disclosure of ‘financial’, ‘service: teaching’ and ‘service: research’ information. Most noticeable was the reduction in information about the academic segments of universities (disciplines, faculties, etc.). On average, no change occurred in the reporting of ‘overview’ information and ‘community service’, although this and the other averages conceal compensating changes, such as those evident from Appendix B for ‘community service’ as reported by the two universities shown there.

15 Auckland University of Technology (AUT) became a university in January 2000, having previously been classed as a polytechnic known as Auckland Institute of Technology (AIT). 16 MAD-scores were also calculated for 1996 and 2000 and compared with PAI results. The Spearman ranking test showed significant correlations between the 1996 MAD-score and PAI numbers at the 99% confidence level, and between the 2000 MAD-score and PAI numbers at the 95% confidence level. Full details are available as per from the first author and at the website: www.mngt.waikato.ac.nz/tears 17 The 2000 NZ university reports were published on average 16 weeks after balance date, with the range being 10–22 weeks. The equivalents for 1996 were 19 weeks on average, and a range of 14–23 weeks.

94

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

5.2. Two loose ends First, the issue of whether an index developed in the 1990s reporting era is applicable to the 1980s era is critical to the validity and reliability of the data in Table 2. We believe that the PAI is relevant in the way we have used it; that is, to see how reporting has changed from a time when aspirations were expressed about improving reporting and accountability (NZSA, 1987). But it would be inappropriate to use these same data to judge and castigate report preparers in that era, as they were doing what was asked and expected of them at the time. Second, the issue of using weightings is referred to above. Now that we have reported the data using the PAI weightings, we can also report about the sensitivity analysis we conducted. This involved recalculating the index numbers with (i) items and categories unweighted, but retaining the polychotomous item scoring, and (ii) items and categories unweighted, and dichotomous scoring. The results support the views of Spero (1979) and Cooke (1989)5 that weighting of items and categories makes little difference to the results. Spearman’s rho calculation comparing PAI with (i) above showed a significant correlation at the 99% confidence level for both 1996 (correlation coefficient 0.970 and significance (two-tailed) 0.000) and 2000 (0.857 and 0.007). These results suggest that for practical purposes the use of weightings in a disclosure index is unnecessary. Theoretically, however, for reasons given earlier, we recommend weighting of both items and categories as intuitively appropriate. Emphasising the importance of this point, if two reports are the same except that one omits, say, the three highest weighted items and the other, the three lowest weighted items, they would score identically on an unweighted index, despite common sense suggesting that the latter report should score higher. The use of dichotomous scoring does make a difference, as Spearman’s rho calculation comparing PAI with (ii) above shows no correlation in either 1996 (0.049 and 0.908) or 2000 (0.333 and 0.420). We believe this difference is important, hence our move from our original dichotomous index to the AD-score and its successors, and so for reasons given earlier, we recommend using a polychotomous scoring scale not only as intuitively appropriate but also as essential to achieving interval or ratio measurement and index numbers that are suitable for use in parametric statistical analyses.

6. Conclusion We stressed the longitudinal nature of this study in Section 1. Particularly interesting is how the PAI builds on the predecessor MAD-score index and attempts to overcome some of the latter’s shortcomings. The similarity of the PAI results to the MAD-scores when applied to the 1996 and 2000 New Zealand university annual reports could indicate that the additional refinement was unnecessary. Such a viewpoint probably overestimates the quality and value of the MAD-score rather than the necessity for PAI. PAI has been crafted from a theoretically derived normative model that was subject to scrutiny and modification by stakeholders and preparers in the delphi exercise. PAI is more comprehensive and includes 58 items compared with 26 in the MAD-score. PAI captures

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

95

timeliness, a report quality regarded as extremely important by university stakeholders. At the extreme, if two reports were compared and one included only the 26 items in the MADscore whilst the other included all 58 items in PAI then, under the MAD-score system both would score identically, assuming common quality of individual disclosures. This would be misleading. Using the PAI, the latter report would rate more highly. Incorporating a polychotomous approach with a zero to infinity scale to assess items, means that the PAI is superior to its predecessors because it generates index numbers that have interval (or even ratio) properties and that can be used in parametric statistical analyses. As to this study’s limitations, we acknowledge that researcher discretion and judgement have been used in constructing the index. However, we believe that PAI is more reflective of stakeholders’ opinions than its predecessors, stemming as it does from the use of stakeholder delphi panels. Moreover, despite the use of criteria benchmarks representing ‘standard’ disclosure in evaluating reports, evaluator discretion comes into play in employing the index, in that, like most other disclosure indices, someone must decide whether particular disclosures are present and how they compare with similar disclosures in other reports. An important aspect of this is that constructors/users of an index should reveal the basis on which judgement has been exercised, including, in this case, that we are coming from a public accountability perspective on university reporting. In both constructing and using the index, we have tried to apply our judgement in ways faithful to this perspective. The final arbiter on the value of the PAI is whether other interested parties accept it and, indeed, whether it influences the compilation and distribution of future reports. Another issue is whether the PAI developed for New Zealand universities could be used for assessing the quality of university annual reports in other countries. On the whole, the disclosures and qualitative characteristics that are important to New Zealand universities’ reports are probably common to universities elsewhere. However, there are some issues included in PAI that are special to New Zealand universities, such as student access and the progress of under-represented groups (e.g. Maori, Pacific Islanders) which may not be relevant to overseas’ universities. The PAI is therefore specific to New Zealand conditions, and some modification would be needed to apply it to university reports in other countries. Another question relates to whether the PAI methodology may be used to construct disclosure indices to evaluate reports issued by other parts of the public sector in New Zealand and other countries, such as local government, state owned enterprises and the health sector. An index for other parts of the public sector would need to be constructed using disclosure items that relate specifically to these other sectors. However, the PAI construction principles described and explained in this paper provide a methodology and framework for application to other sectors. It would be valuable to test these principles by constructing indices of public accountability for other parts of the public sector in New Zealand and in other countries to gauge their generic value. As far as applicability in the private sector and the so-called third sector, there are several generic features that may be useful for researchers to reflect upon, in particular where they intend taking a stakeholder approach to measuring annual reports of groups of organisations from these sectors.

96

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Acknowledgements The authors acknowledge the contributions of the 39 university stakeholders who participated in the delphi study that underpins this research. The paper has benefited from the helpful suggestions of Graham Francis, delegates at the 2002 annual conferences of the British and European Accounting Associations and two anonymous reviewers. As ever, full responsibility rests with the authors.

Appendix A. Summary delphi exercise results

Report qualitative characteristic or disclosure item

Importance (range 0 –3 or 1– 5)a

Report qualitative characteristics Relevance Timeliness Accessibility

4.351 4.057 3.540

Overview: annual report Auditors’ report Statement of accounting policies Statement of managerial responsibility Brief summary (within annual report) Directory information (e.g. table of contents, officials’ names and titles, etc.) Brief summary (as a separate document)

2.663 2.641 2.561 2.165 2.121 1.320

Overview: university Statement of objectives Financial review (narrative) Key facts and figures (statistical highlights) Descriptive report/general review (narrative) Prospective information Students’ review and report Chancellor’s report (i.e. in addition to Vice-Chancellor’s report) Service: community Further education services provided in the community Description of national community service Description of local community service Description of international community service Information on ‘town and gown’ relationships

2.972 2.791 2.564 2.547 2.088 1.771 1.597

1.764 1.502 1.438 1.418 1.345 (continued on next page)

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

97

Appendix A (continued) Report qualitative characteristic or disclosure item

Importance (range 0 –3 or 1– 5)a

Whether community services were requested for the community or volunteered by university

1.283

Service: general Achievements against objectives and targets Statement of service performance Interpretive comment (narrative explanation of service performance) Analysis of student services Staff training and development Equal employment opportunity information Health and safety information Environmental related information No. of childcare spaces per 100 EFTS Qualification of general staff Quality of applicants for positions

2.826 2.630 2.524 2.167 2.167 2.073 1.784 1.692 1.385 1.278 0.950

Service: teaching Input Student numbers (enrolments and EFTS; total and field of study) Tuition fees as % of total revenue (total and field of study) $ Value of operating grants from government $ Value of operating grants from other sources Demand for new places compared with previous year No. of new places created compared with previous year New students by programme area No. and proportion of designated equity groups by programme No. of qualified admitted students unable to enrol in preferred study No. of qualified students not admitted Actual enrolment relative to demand for new places Qualification of student intake Total school graduates at university as % of total school graduates Total EFTS as ratio of 18– 24 age cohort No. of places available in career programs as % of all programs Teaching: process Student: staff ratio (total and field of study) Processes in place to ensure quality of teaching programmes Cost per EFTS (total and field of study)

2.899 2.615 2.540 2.481 2.170 2.078 1.856 1.793 1.769 1.763 1.759 1.727 1.296 1.152 1.120

2.719 2.564 2.515 (continued on next page)

98

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Appendix A (continued) Report qualitative characteristic or disclosure item

Importance (range 0 –3 or 1– 5)a

Percentage of students in full-time/part-time/external courses Library expenditure as % of operating expenditure and $ per EFTS Percentage of students in higher degrees No. of PG fields of study/courses No. of UG fields of study/courses Student services expenditure per EFTS Library stock All staff (total/equity groups) Teaching space (including teaching labs) per EFTS Academic staff (seniority/tenured/qualification/equity groups) Teaching space (including teaching labs) as % of total space Financial aid to students awarded Proportion of staff with PhD No. of beds per 100 EFTS

2.399 2.261

Teaching: output/outcome Expenditure per EFTS (total and field of study) No. and % of EFTS completing their program of study (total and field of study) No. and % of students continuing to further study; and equity Equity students graduating as % of total graduates (total and field of study) Honours degrees awarded as % of total degrees No. and % of graduates employed (total and field of study, and equity) Measure of student satisfaction with programme at time of completion Percentage of EFTS completing each year prior to graduation Expenditure per EFTS per year (total and field of study) Student unit pass rate Reports of external reviews of departments Some ‘impact’ criteria and measures should be developed Measure of student satisfaction at intermediate stages Crude drop-out rate Percentage of equity students in each academic year No. of UG courses completed/100 EFTS Employer satisfaction with former students Average time to complete programme

2.230 2.189 2.126 2.107 1.990 1.946 1.747 1.704 1.691 1.563 1.334 1.070 2.552 2.316 2.008 1.996 1.945 1.931 1.875 1.809 1.713 1.653 1.595 1.518 1.462 1.438 1.336 1.323 1.317 1.288 (continued on next page)

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

99

Appendix A (continued) Report qualitative characteristic or disclosure item

Importance (range 0 –3 or 1– 5)a

No. of equity students at intermediate stages as % of original Former student satisfaction at intervals after graduating (2 years, 5 years, etc.) Expenditure per completed programme year by student Value added (input and quality adjusted output measure) Rate of return (quality adjusted productivity measure) No. of UG courses completed/academic staff Ranking of graduates on standardised scoring system (e.g. GMAT)

1.262 1.234

Service: research No. of masters and PhD students (total and field of study) Research income (grants and contracts) (total $/$ per academic and field of study) No. of research grants and fellowships No. of masters and PhD graduates (total and field of study) Industry contribution to research $ Publications/patents/copyrights (total and field of study) No. of research and sponsored students (total and field of study) Reports of external reviews of departments Quality of research produced by staff Conference presentations (total and field of study) Overall research rating Quality of research produced by students Graduates’ views on the relevance of their degrees to their job and market availability Distinction between public good and basic research, and emphasis on each by the university Destination of research graduates Citations Financial Operating statement Comparative figures for previous year Balance sheet Statement of cash flows Statement of commitments Total value of estates (land, buildings, improvements) Statement of contingent liabilities Statement of cost of services Investments

1.180 1.112 1.076 1.044 0.736 2.369 2.288 2.265 2.237 2.113 2.025 1.871 1.681 1.617 1.611 1.600 1.592 1.540 1.412 1.333 1.124

2.776 2.712 2.692 2.691 2.633 2.598 2.572 2.558 2.541 (continued on next page)

100

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Appendix A (continued) Report qualitative characteristic or disclosure item

Importance (range 0 –3 or 1– 5)a

Financial ratios (i.e. the common accounting ratios) Depreciation Statement of movements in equity (i.e. owners’ equity) Budget information Narrative information about major budget variances Overhead allocation Total employee remuneration costs Cost of teaching Cost of research (NB. cost per EFTS combines teaching and research costs) Cost of compliance with govt. requirements Movements in physical assets Details of loans and lines of credit Loss/gain on sale of assets Rental and leasing costs Detailed financial and performance information at Faculty level

2.506 2.452 2.441 2.388 2.341 2.320 2.028 2.011 1.889 1.740 1.722 1.629 1.528 1.514 1.511

EFTS: equivalent full-time students. Mean scores for ‘importance’ as rated by stakeholders on a 0–3 Likert scale where 0: should not be disclosed, 1: should be disclosed but item is of minor importance, 2: intermediate importance, 3: extremely important. Opinions about relevance and timeliness were collected on a 1–5 Likert scale where 1: minor importance and 5: extremely important. Opinions about Accessibility are scored where 1: ‘distributed to University Council, Ministry of Education and Parliament only’ to 5: ‘reports are distributed to all major stakeholder groups including students and other users of university services’. a

Appendix B. Sample public accountability index scores for two universities’ 1996 and 2000 reports

Item categories

Category weightings

University of Auckland 1996

Category scores Timeliness Overview: report Overview: university

10 7 10

0 94 68

2000

Victoria University of Wellington 1996 2000

100 80 140 90 86 86 77 86 77 (continued on next page)

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

101

Appendix B (continued) Item categories

Category weightings

University of Auckland 1996

Financial Service: general Service: teaching Service: research Service: community service Public accountability index score Index items

25 8 25 13 2 Item weightings

76 51 43 48 54 54

2000 76 80 44 67 84 71

University of Auckland

Victoria University of Wellington 1996 2000 85 80 87 82 86 84

51 74 42 22 60 61

Victoria University of Wellington 1996 2000

1996

2000

100

0

5

4

7

Overview: report Auditors’ report Statement of accounting policies Directory information Statement of managerial responsibility Brief summary

25 25 20 20 10

5 5 6 5 0

5 5 5 5 0

5 5 4 5 0

5 5 5 4 0

Overview: university Statement of objectives Descriptive report/general review Financial review Key facts and figures Prospective information

35 20 20 15 10

4 5 3 2 1

4 5 2 5 3

5 5 4 5 0

4 4 4 5 1

Financial items Financial performance statement Statement of cash flows Statement of cost of services Financial position statement Budget information Overhead allocation Depreciation Financial ratios Investments Total value of estates

15 15 15 12 10 10 5 5 5 5

4 4 4 3 4 5 2 3 4 3

5 4 4 4 5 2 4 1 4 3 (continued

Detailed scores Timeliness

5 3 5 4 5 0 4 4 4 3 5 0 1 2 0 0 5 5 3 4 on next page)

102

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Appendix B (continued) Index items

Item weightings

University of Auckland

Victoria University of Wellington 1996 2000

1996

2000

3

4

2

5

5

Service: general Achievements vs. objectives and targets Interpretive comment Analysis of campus services Staff training and development Equal employment opportunity info. Environmental related information Health and safety information

30 20 15 15 10 5 5

3 3 2 3 3 0 0

6 5 4 2 3 0 0

5 4 5 3 5 0 0

5 3 5 3 4 0 0

Service: teaching Input (of students and resources) Student numbers Cost per EFTS Revenues Staff Equity group information Qualification of student intake Space Financial aid

12 7 7 6 3 3 3 2

4 1 1 2 3 0 1 2

4 2 3 3 4 1 4 1

5 5 5 5 3 0 3 3

3 0 2 3 5 0 4 3

Service: teaching Process Student: staff ratios Processes to ensure quality of teaching Library service information Computer service information Fields of study

12 8 6 4 4

2 2 4 1 2

0 2 4 0 3

5 5 6 3 5

0 5 2 1 2

5 4 3 3 3 2 2

5 3 2 0 0 0 3

5 1 3 0 0 0 0 (continued

Commitments and contingencies

Service: teaching Output/outcomes Graduates Destination of students Departmental reviews Pass rates Student satisfaction Employer satisfaction UG courses completed/100 EFTS

6 2 4 4 2 2 6 0 4 5 0 0 0 0 on next page)

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

103

Appendix B (continued) Index items

Item weightings

University of Auckland

Victoria University of Wellington 1996 2000

1996

2000

1

0

0

0

0

Service: research Graduates Postgraduate students Research income Publications Destination of research graduates

25 25 25 20 5

3 3 2 2 0

3 4 4 3 0

5 6 3 3 0

0 0 2 3 0

Community service Local community service National community service International community service

40 30 30

3 3 2

3 5 5

4 4 5

3 3 3

Average time to complete programme

Appendix C. Criteria for a sample of the public accountability index items9 Disclosures required to meet ‘benchmark’ criteria ( ¼ a score of 5) on the Public Accountability Index. Timeliness 1. Time elapsing between balance date and report publication Report available to the public in 12– 13.6 weeks from balance date (score ¼ 5). Score increases by 1 for every two weeks or part of two weeks earlier than 12 weeks, and decreases by 1 for every two weeks or part of two weeks later than 13.6 weeks, and so a report available 22 weeks or more after balance date scores 0 for timeliness (8 –9 weeks ¼ 7; 10 – 11 ¼ 6; 12 – 13 ¼ 5; 14 – 15 ¼ 4; 16 – 17 ¼ 3; 18 – 19 ¼ 2; 20 – 21 ¼ 1). Overview: report 2. Auditors’ Report Full audit report as required by statute. Clearly titled, statement that auditor has had access to all information required, basis of audit opinion, audit opinion, name of auditor and date audit signed. Overview: university 7. Statement of objectives Separate statement that must include the following items: mission, goals and performance targets in specific, concise, understandable and realistic terminology. The items must have measurable/quantitative dimensions and a given time frame.

104

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Financial 13. Financial performance statement A summary statement(s) by input items (e.g. salaries, consumables), or organisational unit (e.g. faculty, school) or programme (e.g. course of study, research discipline); detailed disclosure with informative footnotes, one year comparative and subheadings and other aids to understanding. Includes all expenses (e.g. depreciation, leave accruals) and revenues. Not more than 10% of total expenses in any single unanalysed item. Service: general 24. Achievements vs. objectives and targets Clear specification of objectives or targets especially in respect of teaching and research service, and achievements in respect of each. Quantified where possible in respect of inputs, process, costs and outputs. Comparisons between objectives/targets and achievements should be made for at least 10 elements of service performance. Service: teaching Input (of students and resources) 31. Student numbers Number of students and EFTS based information both in aggregate for institution and by faculty/school. Section title, at least 4-year trend and including illustration(s). Full-time, part-time and extramural breakdowns; undergraduate, postgraduate, diploma and other courses as appropriate. Service: research 52. Graduates Number of graduates at the postgraduate level, in total and by discipline and course of study. 4-year trends and illustrations. Community service 57. Local community service Descriptive, such as public lectures, advisory and affiliation roles, town and gown relationships, other academic, cultural and artistic services provided.

References Ahmed, K., Courtis, J.K., 1999. Associations between corporate characteristics and disclosure levels in annual reports: a meta-analysis. The British Accounting Review 31, 35–61. Botosan, C.A., 1997. Disclosure level and the cost of equity capital. The Accounting Review 72, 323 –349. Cameron, J., Guthrie, J., 1993. External annual reporting by an Australian university: changing patterns. Financial Accountability and Management 9, 1–15. Cave, M., Hanney, S., Kogan, M., 1991. The Use of Performance Indicators in Higher Education, Jessica Kingsley, London. Chen, R.S., 1975. Social and financial stewardship. The Accounting Review 50, 533–543. Committee of Vice-Chancellors and Principals, Committee of Scottish Higher Education Principals and Standing Conference of Principals Limited, 1994. Statement of Recommended Practice: Accounting in Higher Education Institutions, Committee of Vice-Chancellors and Principals, Committee of Scottish Higher Education Principals and Standing Conference of Principals Limited, London. Cooke, T.E., 1989. Disclosure in the corporate annual reports of Swedish companies. Accounting and Business Research 19 (74), 113–124.

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

105

Coy, D., Dixon, K., 1996. Annual reporting continues improvement. Chartered Accountants’ Journal 75 (10), 3839. Coy, D., Pratt, M., 1998. An insight into accountability and politics in universities: a case study. Accounting, Auditing and Accountability Journal 11, 540 –561. Coy, D., Tower, G., Dixon, K., 1991. Tertiary education in New Zealand: radical changes to funding and accountability. Journal of Tertiary Education Administration 13, 83–93. Coy, D., Tower, G., Dixon, K., 1993. Quantifying the quality of tertiary education annual reports. Accounting and Finance 33, 121–129. Coy, D., Tower, G., Dixon, K., 1994. Public sector reform in New Zealand: the progress of tertiary education annual reports, 1990–92. Financial Accountability and Management 10, 253–261. Coy, D., Dixon, K., Buchanan, J., Tower, G., 1997. Recipients of public sector annual reports: theory and an empirical study compared. The British Accounting Review 29, 103–127. Coy, D., Fischer, M., Gordon, T., 2001. Public accountability: a new paradigm for college and university annual reports. Critical Perspectives on Accounting 12, 1–31. Cutt, J., Trotter, L., Lee, C., 1993. Performance measurement and accountability in Canadian universities. Financial Accountability and Management 9, 255 –266. Davis, S.W., Menon, K., Morgan, G., 1982. The images that have shaped accounting theory. Accounting, Organizations and Society 7, 307 –318. Department of Employment, Education and Training, 1994. How Australian universities perform: the DEET tables. Campus Review 26 May. Dixon, K., Coy, D., Tower, G., 1991. External reporting by New Zealand universities, 1985– 89: improving accountability. Financial Accountability and Management 7, 159 –178. Dixon, K., Coy, D., Buchanan, J., Fry, J., 2002. Annual reports of social enterprises: an audience perspective of their qualitative characteristics’ [Abstract], European Accounting Association 25th Annual Congress: Programme and Collected Abstracts, European Accounting Association, Copenhagen, p. 108. Education Act No. 80 of 1989. Engstrom, J.H., 1988. Information needs of college and university decision makers, Research Report, Governmental Accounting Standards Board, Norwalk, CT. Fischer, M., Gordon, T., 1991. Towards better annual reports for colleges and universities. Government Accountants’ Journal 40 (2), 35–44. Gannicott, K., 1994. Quality: the Gannicott table. Campus Review 9 (June), 8–9. Gray, R., Haslam, J., 1990. External reporting by UK universities: an exploratory study of accounting change. Financial Accountability and Management 6, 51– 72. Ingram, R.W., Robbins, W.A., 1992. A partial validation of the GASB user needs survey: a methodological note. Research in Governmental and Non-profit Accounting 7, 41–52. Institute of Chartered Accountants of New Zealand, 2002. TPA-9: Service Performance Reporting: Technical Practice Aid Number 9, Institute of Chartered Accountants of New Zealand, Wellington. Johnes, J., Taylor, J., 1990. Performance Indicators in Higher Education, The Society for Research into Higher Education and Open University Press, Buckingham. Jones, D.B., Scott, R.B., Kimbro, L., Ingram, R., 1985. The Needs of Users of Governmental Financial Reports, Governmental Accounting Standards Board, Stamford, CT. Linstone, H., Turoff, M. (Eds.), 1975. The Delphi Method, Techniques and Applications, Addison-Wesley, Reading, MA. Marston, C.L., Shrives, P.J., 1991. The use of disclosure indices in accounting research: a review article. The British Accounting Review 23, 195 –210. McCulloch, B.W., Ball, I., 1992. Accounting in the context of public sector management reform. Financial Accountability and Management 8, 7–12. Ministry of Education, 1991. Financial Reporting for Tertiary Institutions, Ministry of Education, Wellington. Mintzberg, H., 1987. Crafting strategy. The Harvard Business Review 87 (4), 66–75. New Zealand Society of Accountants, 1987. Statement of Public Sector Accounting Concepts, New Zealand Society of Accountants, Wellington. New Zealand Society of Accountants, 1993. Statement of Concepts for General Purpose Financial Reporting, New Zealand Society of Accountants, Wellington.

106

D. Coy, K. Dixon / The British Accounting Review 36 (2004) 79–106

Normanton, E.L., 1971. Public accountability and audit: a reconnaissance. In: Smith, B.R., Hague, D.C. (Eds.), The Dilemma of Accountability in Modern Government: Independence versus Control, Macmillan, London, pp. 311 –345. Patton, M.Q., 1990. Qualitative Evaluation and Research Methods, second ed., Sage, Newbury Park, CA. Performance Indicators Task Force, 1989. Performance Indicators for Tertiary Institutions, Performance Indicators Task Force, Wellington. Pettigrew, A.M., 1985. Contextualist research: a natural way to link theory and practice. In: Lawler, E.E. III, Mohrman, A.M. Jr., Mohrman, S.A., Ledford, G.E. Jr., Cummings, T.G. (Eds.), Doing Research that is Useful for Theory and Practice, Jossey-Bass, San Francisco, CA, pp. 222– 248. Public Finance Act No. 44 of 1989. Scott, D.R., 1941. The basis for accounting principles. The Accounting Review 16, 341–349. Siegel, G., 1993. Improving the quality in data collection in behavioral research: the use of interval and ratio scales, Proceedings of the Fifth Asian-Pacific Conference on International Accounting Issues, California State University, Mexico City, pp. 118 –121. Silverman, D., 1985. Qualitative Methodology and Sociology, Gower, Aldershot, Hants. Silverman, D., Gubrium, J.F., 1989. Introduction. In: Gubrium, J.F., Silverman, D. (Eds.), The Politics of Field Research: Sociology Beyond Enlightenment, Sage, London, pp. 1 –12. Singleton, W.R., Globerman, S., 2002. The changing nature of financial disclosure in Japan. The International Journal of Accounting 37, 95 –111. Spero, L.L., 1979. The extent and causes of voluntary disclosure of financial information in three European capital markets: an exploratory study, Unpublished Doctoral Thesis, Graduate School of Business, Harvard University. Tower, G., Coy, D., 1990. Urgent need to end crisis in academia. The National Business Review 29, 13. Turner, R., 1989. Deconstructing the field. In: Gubrium, J.F., Silverman, D. (Eds.), The Politics of Field Research: Sociology Beyond Enlightenment, Sage, London, pp. 13 –29.