- Email: [email protected]
How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums
Accepted Manuscript
How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums Antonella Basso, Francesco Casarin, Stefania Funari PII: DOI: Reference:
S0305-0483(16)30580-1 10.1016/j.omega.2017.09.010 OME 1836
To appear in:
Omega
Received date: Revised date: Accepted date:
7 September 2016 23 August 2017 30 September 2017
Please cite this article as: Antonella Basso, Francesco Casarin, Stefania Funari, How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums, Omega (2017), doi: 10.1016/j.omega.2017.09.010
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • A two-stage model with weights restrictions measures the performance of museums • This is the first model that exploits an integrated DEA-BSC approach for museums
CR IP T
• In the first stage the museums performance for each BSC perspective is evaluated • In the second stage the overall performance score is computed
AC
CE
PT
ED
M
AN US
• An empirical analysis to Venice MUVE museums shows the applicability of the model
1
ACCEPTED MANUSCRIPT
How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums Antonella Bassoa , Francesco Casarinb , Stefania Funaric a
CR IP T
Department of Economics, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected] b Department of Management, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected] c Department of Management, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected]
Abstract
CE
PT
ED
M
AN US
Measuring the performance has become an important issue also in the cultural sector. In the recent literature, in order to evaluate the performance of museums, Data Envelopment Analysis (DEA) models have been proposed; these models allow to take into account the multidimensional nature of the museum performance. In this paper we exploit the capability of DEA to assess the performance by integrating the DEA approach with the Balanced Scorecard (BSC) tool; this is the first study that proposes a joint application of DEA and BSC to museums. To this aim we propose a new two-stage DEA-BSC approach. In the first stage we set a Balanced Scorecard scheme devised for museums, then we define a proper DEA model for each BSC perspective and compute the DEA efficiency score for every perspective. In the second stage we define a further DEA model which combines the efficiency scores of the various BSC perspectives into an overall performance indicator. In order to prevent the computation of the efficiency score to be almost exclusively determined by one output while disregarding the others, we set proportional restrictions on the virtual outputs. Finally, the DEA-BSC joint approach is applied to the municipal museums of Venice, carrying out a detailed empirical analysis.
AC
Keywords: Performance evaluation, Data Envelopment Analysis, Balanced Scorecard, museums, virtual weights restrictions 1. Introduction This contribution intends to evaluate the performance of museums through the development of an analytical model based on the joint use of two measuring tools: Balanced Scorecard (BSC) and Data Envelopment Analysis (DEA). The model proposed may be used to allocate public or private resources to the most efficient museums and decide the amount of resources to allocate to each of them. The evaluation of the efficiency of each museum allows, in addition, the Preprint submitted to Omega
October 6, 2017
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
identification of efficient benchmarks and the dissemination of the best practices in the museum network of reference. The production of museum services is facing multiple challenges, some external and other internal to the organisation. The main external challenges for the museums management arise from the economic crisis; specifically, we observe both a reduction in the public funding for arts and a change in the consumption of culture due to a decrease in the personal income of many agents. Moreover, museums sponsors have changed their attitude towards funding and ask for a more precise and proactive assessment of the actual returns that can be achieved; even donors request a more detailed planning and information on supported activities. As a result, museums are increasingly coping with competitive situations, vying to attract both users and donors, competing with many organisations operating in the field of the free time market. From an internal point of view, the challenge for museums primarily concerns the pursuit of a clearly defined mission, focused on their social purposes. This mission must be translated into targets that are consistent not only with the social effectiveness but also with the managerial efficiency. Museums, as well as other cultural organisations (Niven [34]), may respond to the external and internal challenges by implementing suitable tools of performance measurement. Performance measurement systems aim at providing synthetic information about the efficiency and effectiveness of services and at enabling comparative assessments which are transparent, simplified and standardized. However, though performance is a multidimensional concept, the traditional systems of performance measurement tend to focus mainly on the economic-financial side and rely on balancesheet indicators to analyse the corporate performance. It has been critically remarked that the economic-financial side alone neglects to take into account other resources that are fundamental for business development, such as the skills of the staff, the trust relationship with customers and the culture of innovation (Kaplan and Norton [27]). In museums, as in the other nonprofit organisations, these problems are even worse, because of the higher level of transparency required by the presence of public funding and donors. Indeed, the (often prevailing) public funding of these organisations requires a strong need for control tools that are able to highlight and possibly measure the social impact of the activities. The combination of the assessment problems arising from the pursuit of the institutional and social purposes and the typical problems of traditional control systems (Carnegie and Wolnizer [9]) makes the control of the performance in museums particularly complex. The design of a tool for the performance measurement of museums that overcomes the drawbacks of the traditional control systems and takes into account also the social aspects is still an open issue, although there are a few contributions that discuss it (Paulus [35], Marcon [32]). In this paper we propose a quantitative model that provides a tool that measures the museums performance with a balanced approach which is able to take into consideration the efficiency of different aspects of the museum activities. In this model we combine two methodologies taken from different fields, BSC (Kaplan and Norton [27] and Kaplan [24]) and DEA (Charnes et al. [11]), 3
ACCEPTED MANUSCRIPT
CE
PT
ED
M
AN US
CR IP T
with the aim to compare the performance of a set of museums. While there is a substantial literature on BSC and DEA, and there is a number of studies that integrate the two approaches (for a review see Amado et al. [2]), there are few studies on the application of BSC to museums (see Table 1 in Section 2) and on the application of DEA to museums (see Table 2 in Section 3); on the other hand, to our knowledge this is the first study that proposes a joint application of DEA and BSC to museums. The model is organized in two stages. In the first stage we compute a DEA efficiency score for each BSC perspective, applying a Balanced Scorecard scheme specifically devised for museums. In the second stage we define a synthetic DEA model which combines the efficiency scores of the various BSC perspectives into a comprehensive performance indicator. In order to prevent the computation of the efficiency score of a museum to be almost exclusively determined by one output variable while disregarding the presence of other important outputs, we set proportional restrictions on the virtual outputs. In our experience, such an approach can be welcomed by the museum management and may be useful to take into consideration some value judgements in the evaluation process. In order to take into account the sustainability efforts undertaken by museums, we also propose a sustainability indicator for museum institutions. In the final part of the paper the models proposed are applied to a real case study in order to evaluate the performance of the museums managed by the Venice Municipal Museums Foundation (MUVE, Fondazione Musei Civici di Venezia). The empirical analysis shows that the models proposed can be effectively implemented to compare the performance of a set of museums. The structure of the paper is as follows. Sections 2 and 3 discuss the literature on the use of the Balanced Scorecard and DEA approaches for museums management and briefly present a variable returns-to-scale (VRS) DEA model for museums. The new two-stage DEA-BSC model is proposed in Section 4, where a sustainability indicator for museums is also suggested. The proportional restrictions on the virtual outputs are discussed and included in the DEA models in Section 5. Section 6 presents the data and the results of the empirical analysis. Finally, Section 7 summarises some conclusions.
AC
2. Balanced Scorecard for museum management The Balance Scorecard is a framework that links the performance of an organisation to its strategy and emphasises the connections between strategy, actions and results (Kaplan and Norton [26]). According to its original formulation, BSC has been proposed for the evaluation of profit-oriented organisations. The BSC model aims to: create a system of quantitative information on critical variables of success; counterbalance the economicfinancial data with a more complete analysis of the environment, both inside and outside the organisation, considering shareholders and customers; overcome barriers – related to the vision, the people, the ways of resources allocation, the commitment 4
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
of the management – that prevent from aligning the implementation of the strategy to its objectives. In general, BSC focuses on four perspectives according to which the organisation is evaluated: the Financial, the Customer, the Internal process, the Innovation and learning perspectives. The original formulation provides for each perspective the institutional goals and the specific objectives, the performance indicators, the target levels of the indicators and the actions to be taken in order to achieve the targets chosen. The indicators of the economic-financial (or simply “Financial”) perspective reflect the profitability (ROI, cash flow, operating income, etc.) and the ability to meet the needs of the shareholders. The Customer perspective refers to the way in which the company should be perceived by customers to be able to realise its vision; examples of indicators for this perspective are loyalty, customer satisfaction, the number of new customers, etc. From the Internal process point of view, the company explicits what must be done internally in an excellent manner to meet the expectations of shareholders and customers. Examples of indicators are the quality of the booking services and the information systems, the costs of the service production, the ability to differentiate the product. Finally, the Innovation and learning perspective can be interpreted as the ability to develop continuous improvement, innovation and learning in order to deal successfully on the medium and long term. This perspective relates to the investments aimed at increasing the capacity of systems, processes and human resources. Investments in training, equipment, the ability to increase empowerment, the staff satisfaction are examples of useful indicators. In the original framework and in many applications in the BSC literature, the basic mechanism of BSC relies on the assumption that the four perspectives are connected together by cause-effect relations, forming a system. The Innovation and learning perspective lies at the bottom of the system, and influences the Internal process perspective (quality and timing of processes); this in turn affects the Customer perspective, which in turn leads to the final Financial perspective, positioned at the top of the chain. At the centre of the system there is the strategy deriving from the vision of the management. The application of the Balanced Scorecard to nonprofit organisations, and therefore also to museums, however, faces a basic problem, which causes BSC to require some adjustments with respect to the original setting. The basic problem posed by nonprofit organisations to BSC concerns the fulfilment of the social purpose of their activity: the social purpose is usually directed toward multiple primary stakeholders groups, not only toward shareholders. Categories bearers of institutional interests, for example different groups of citizens, the local community, the public administration, even donors, appear to be stakeholders to whom the nonprofit organisation may address its actions. Moreover, the social purpose entails two further difficulties: to identify the types of primary stakeholders (see Kaplan and Norton [27]) and to understand how the social purpose is reflected in the cause-and-effect relationships between the perspec5
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
tives. Adopting an incomplete-contract view, Speckbacher [45] argues that “there is no clearly defined stakeholder group with homogeneous expectations and objectives that can be placed at the top of the hierarchy”. Speckbacher [45] also points out that while in the standard BSC the cause-and-effect chains are unidirectional, in nonprofits a multidirectional structure of cycles is typical. For nonprofit organisations, therefore, the original BSC hierarchy is challenged, and may even be arguable, given the multiple nature of their stakeholders (Norreklit [33]). Some contributions in the literature have tried to take into account these issues. According to Kaplan and Norton [27] the original BSC framework has to be adapted in order to be applied to nonprofit organisations. They suggest to place an overarching objective, representing the mission of the nonprofit organisation itself, at the top of the framework. The long-term mission should orient the BSC objectives while the Financial perspective is no longer placed at the top of the hierarchy since the financial success is not the primary objective of these organisations. Along these lines, the Financial perspective is moved from the top to the bottom line (Kaplan [23]), so that the initial financial boost is transmitted along the different perspectives until the final perspective Customer. Kaplan and Campbell [25] describe the application of this framework to the Boston Lyric Opera. Weinstein and Bukovinsky [48] apply the same scheme in a further study on the Boston Lyric Opera and underline the key role played by the employees and by the commitment of the lower hierarchical levels of the organisation in the collection of information required by BSC. On the other hand, assuming a relational and postmodern point of view on arts, Boorsma and Chiaravalloti [8] emphasize the role of the mission and of the stakeholders and develop a new version of the BSC framework, suitable for the artistic-mission led organisations. Instead of following the indication of Kaplan and Norton to consider the mission as a over-arching long-term objective and to move the Financial perspective to the bottom line of the chain, they place the mission and the stakeholder relationships on the top of the cause-effect chain and explicitly add the mission to the BSC framework as a further performance dimension. Hence, their BSC model is based on 5 dimensions; the Innovation and learning perspective, instead, is still kept at the bottom line of the chain. Their study is an interesting theoretical contribution that, however, has not been actually implemented. Indeed, the implementation of this model would require the definition of suitable indicators that measure the “different sorts of artistic value created by the arts organisation for the key stakeholder groups” and this may often be difficult. The above mentioned applications show that the scheme proposed by Kaplan and Norton [27] and Kaplan [23] may be conveniently applied to some nonprofit organisations. As for the models proposed for museum organisations, we may find a few contributions in the literature by Marcon [32], Haldma and L¨aa¨ts [22] and Zorloni [50]; with a different aim, namely to propose a disclosure index, we cite also Wei et al. [47]. These papers present different implementations within the BSC framework for the assessment of the performance of museums; for convenience, Table 1 summarises the main features of the BSC models they propose. We may observe that all the three models discussed in Marcon [32], Haldma and L¨aa¨ts [22], Zorloni [50] present a large list of detailed indicators to measure the 6
ACCEPTED MANUSCRIPT
Table 1: Literature on BSC applications to the measurement of the performance of museums.
Approach used
Context of application Perspectives: Stakeholders’
Financial Internal process
Employees and capability of the organisation
Investigations to directly check the degree of satisfaction of the public authorities and private individuals, Income from public subsidies and donations over the total revenues and income, The sum of the resources acquired and invested by the museum x a multiplication factor, Funds attracted from areas outside the region of the museum. Income (tickets, catalogs sold and other material, other services), Number of visitors, Number of complaints, Satisfaction scores obtained through surveys, Assessment of the behavior of the visitors through qualitative methods, Contingency valuations, Indicators of the value attributed to the services by customers/users (transport costs, revenues from access fees, subscriptions to educational services). Analysis of the temporal evolution of the relationship between proceeds/income and operating costs, Temporal evolution of the income to operating costs ratio, Temporal evolution of the ratio between income and expenditure. Number of restorations carried out, Number of publications, Number of items catalogued, Number of loans, Number of exhibitions, Number of guided tours, Average cost per event. Incidence of the cost of training activities on the total operating costs, Number of employees adequately trained, Number of training days per year, Ad hoc surveys, Turnover ratio. Wei, Davey and Coy [47] To examine the reporting practice of leading museums in New Zealand and the UK and to develop a museums’ performance accountability disclosure index. This paper presents a disclosure index that uses a framework based on BSC. The BSC provides a link between internal reporting of key performance issues and the need of a community organisation to report information to the public. Museums, New Zealand and UK
CE
PT
Context of application Perspectives: Institutional and social
ED
Approach used
Customer
Number of items borrowed and photocopies, Number of visiting researchers and readers, Number of study and training courses based on the collections, Number of items deposited, Number of depositing institutions, Number of consultations, Number of publications presenting the collections, Number of lectures presenting collections. Variance in operating costs, Rate variance of department budget as % of budget, Net revenue variance, Fulfilment of cost budget items. Number of units received by collections in comparison to the previous year, Number of correspondents, Number of contributors, Number of senders of images, Number of images and photos, Number of items repaired in the preservation laboratory, Number of controlled ventilated and cleaned items in the collection, Number of archived and preserved items, Number of researched museum items, Percentage of items preserved in the contemporary requirements, Number of units associated with the database, Number of items in the museum. Rate of satisfaction, Number of training courses, Rate of employees with professional certification/qualified/PHD degree, Percentage of employees aware of the museum’s vision and goals, Number of interdepartmental cooperation groups/projects, Percentage of employees supporting managerial decisions. Marcon [32] To analyse theoretically the advantages and the difficulties emerging from the adaptation of BSC to museums. After an analysis of the problems affecting existing BSC models, in the light of the challenges museums are facing, the contribution proposes a framework to adapt BSC to museums. Museums as nonprofit organisations.
M
Objective of the study
CR IP T
Objective of the study
Models description and indicators Haldma and L¨ a¨ ats [22] To examine the impact of the design and implementation of BSC on the performance measurement and management of museums After the presentation of a theoretical framework, covering the BSC implementation issues in public sector organisations and performance measurement of museums, an empirical case study is presented and discussed. Central museum, Estonia
AN US
Main features
Financial
AC
Internal process
Learning and development
Objective of the study Approach used
Context of application
7
Approach used
Context of application Perspectives: Intellectual
Percentage of works displayed, Number of artworks loaned to other museums, Number and quality of institutions to which the museum has loaned artworks, Number of artworks purchased in the previous year, Percentage of permanent collections accessioned and catalogued, Number of scholarly articles published by museum staff, Number of catalogues published, Number of curators with adjunct appointments at universities, Number of talks by curators at scholarly conferences, Number of scholarly symposiums organized by the museum, Number of research grants awarded, Number of positive critical reviews, Weekly media coverage of an exhibition, Percentage of total exhibitions presented that were organized by the museum. Range and variety of programs offered, Increase in audiences cultural knowledge and awareness, Increase in audiences knowledge of a particular author or art movement, Percentage of visitors willing to return, Increase in first-time visitors, Number of schoolchildren visiting the museum, Number of attendees at lectures, Number of artworks shown on the museums web site, Percentage of customers satisfied with programs and services, Number of people participating in educational programs versus all other available options, Number of international curators, Number of exhibitions assembled by the museum slated to travel to other international art museums, Number of collaborative projects with other cultural institutions. Availability of sensitive documents and other materials on the museums web site, Ability to meet fundraising targets, Ability to meet performance budgets, Percentage of the budget dedicated to fundraising, Fundraising efficiency (fundraising expenses divided by total contributions received), Contributed revenue as a percentage of operating revenue, In-kind sponsorships and level of earned revenue. Percentage of goals met from the most recent strategic plan or percentage of employees whose performance plans link to objectives of the strategic plan, Number of full-time curators, Number of fulltime curators with a PhD in art history, Percentage of the budget dedicated to training and professional development, Use of 360-degree feedback to evaluate staff performance, Percentage of retained employees, Percentage of employees who are satisfied with working at the museum (on a survey) and percentage of employees who are satisfied with the quality of internal communication.
and
Fi-
PT
Governance nancial
ED
M
Public
Zorloni [50] To build a model for evaluating the performance of art museums based on BSC methodology. In a first stage, the work presents a theoretical framework for assessing value creation in the museum sector and discusses the use of the BSC framework as a means for framing and focusing the strategic goals and activities of museums. In a second stage, a working set of Critical Success Factors for visual art museums is developed and used to build a framework to assess the performance. Art museums, USA and UK
AN US
Objective of the study
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
Learning and Growth
8
ACCEPTED MANUSCRIPT
AN US
CR IP T
performance with regard to the different perspectives. These indicators are very differentiated among the three models; moreover, some of them require the collection of detailed data, often obtainable only with specific analysis involving surveys or internal and reserved documents. Only Haldma and L¨aa¨ts [22] presents an application to a museum (a central museum in Estonia), while Marcon [32] and Zorloni [50] merely list the indicators they propose. Notice that Marcon [32] adds to the four classical perspectives a fifth one, the Institutional and social perspective, which is placed side by side with the Customer perspective. On the other hand, Haldma and L¨aa¨ts [22] focus the attention on stakeholders, which include customers next to other interested parties (items borrowed and photocopies, visiting researchers and readers, . . . , see Table 1), and centre the classical Innovation and learning perspective on employees (Employees and capability of the organisation perspective). As for Zorloni [50], her definition of the four BSC perspectives somewhat strays from the original BSC model: the Intellectual perspective takes the place of the Innovation and learning perspective, the Public perspective replaces the Customer one, the Governance and Financial perspective replaces the Financial one and the Learning and Growth perspective takes the place of the Internal process one. 3. DEA models to evaluate the performance of museums
AC
CE
PT
ED
M
DEA is a methodology that can be used to measure and compare the performance of homogeneous decision-making units (DMUs). DMUs refer to a group of companies, divisions, departments and administrative units that have the same goals and objectives. DEA analyses the comparative efficiency of DMUs based upon inputs and outputs. From a technical point of view, DEA allows to compute an efficiency score by solving a linear programming problem for every DMU to identify the nonparametric production frontier; this score represents a radial measure of efficiency computed with respect to the estimated efficient frontier (see e.g. Cooper et al. [16]). Originally proposed by Charnes et al. [11], DEA has a wide range of applications, from the performance evaluation of nonprofit institutions such as schools, universities, hospitals, public agencies, to the performance evaluation of banks, insurance companies, and mutual funds (for a quick review see e.g. Liu et al. [29]). Since the DEA methodology provides an overall measure of performance that is simultaneously open to multiple input and multiple output situations, the approaches based on DEA, when applied to the cultural organisations, are able to take into account the multidimensional nature of museum’s performance. Therefore, in the last decade, a growing number of researchers have adopted a DEA approach to assess the relative performance of museums; the main contributions are displayed in Table 2. Among the first contributions we may cite Mairesse and Vanden Eeckaut [31], Pignataro [37], Basso and Funari [6] and [7], while more recent contributions are Del Barrio et al. [17], Taheri and Ansari [44], Carvalho et al. [10], Del Barrio and Herrero [18]. When applied to cultural organisations, the approaches based on DEA allow to:
9
ACCEPTED MANUSCRIPT
Table 2: Literature on DEA applications to the measurement of the performance of museums. Model used
Input and output variables
CCR (CharnesCooper-Rhodes), input oriented BCC (BankerCharnes-Cooper), FDH (free disposal hull)
Basso and Funari [7]
A set of Italian municipal museums
CCR, input oriented BCC, cross efficiency
Carvalho, Silva Costa and Carvalho [10] Del Barrio, Herrero and Sanz [17]
A set of Portuguese museums
BCC
A regional system of Spanish museums
Del Barrio and Herrero [18]
A regional system of Spanish museums
Principal components and cluster analysis, CCR, input oriented BCC and superefficiency CCR, input oriented BCC, Malmquist index
Inputs: number of workers, exhibition area. Outputs: number of visitors paying the full entrance fee, number of visitors paying a reduced fee or admitted free, number of temporary exhibitions, number of other activities carried out. Inputs: number of workers, exhibition area. Outputs: number of visitors paying the full entrance fee, number of visitors paying a reduced fee or admitted free, number of temporary exhibitions, number of other activities carried out. Inputs: number of collaborators, index of facilities, number of days open to the public. Output: number of visitors. Inputs: staff, size and equipment. Output: number of visitors.
Mairesse and Vanden Eeckaut [31]
Museums from the French speaking region of Belgium
Pignataro [37]
A set of Sicilian museums
FDH-CRS (constant returns-to-scale), FDHNIRS (non increasing returns-to-scale), FDH-NDRS (non decreasing returnsto-scale), FDH-RRS (restrictive concept of returns-to-scale) CCR, BCC, Malmquist index
Taheri Ansari [44]
A set of cultural and historical museums in Tehran
CCR, MAJ (Mehrabian, Alirezaee, Jahanshahloo) full ranking method
PT
and
ED
M
Basso and Funari [6]
CR IP T
Context of application A set of Italian municipal museums
AN US
Study
Inputs: employment, size and equipment. Outputs: number of visitors, number of temporary exhibitions, a measure of social impact, percentage of artworks loaned and new acquisitions. Inputs: scientific and technical personnel, operational budget, security services. Outputs: percentage of the collection inventoried, a technical indicator, number of temporary exhibitions, number of publications, number of communication actions, opening hours, number of visitors.
Inputs: administrative and technical staff, custodians, square meters of display space. Output: number of visitors. Inputs: space and accessibility index, human resource index, facility index, introduction index. Outputs: visitors index.
AC
CE
1. take into account the multidimensional nature of museum’s performance, since the methodology provides an overall measure of performance that simultaneously considers a multiple input - multiple output situation; 2. identify a “best practice” set of museums, named peer group, which acts as a benchmark for a museum that has been evaluated as inefficient; 3. allow to identify possible paths towards efficiency.
Let us consider a set {1, 2, ..., n} of museums whose performances have to be evaluated; let us assume that each museum uses m inputs (resources required for its activities) to produce t outputs (services provided by the museum). Let us denote by Eo the efficiency score associated to a given museum o (o ∈ {1, 2, ..., n}), defined as the ratio between the weighted sum of outputs and the weighted sum of inputs. Eo can be obtained, under the assumption of variable returns-to-scale (VRS) and 10
ACCEPTED MANUSCRIPT
an output orientation, by solving the following linear programming problem in dual form, addressed as BCC (Banker-Charnes-Cooper) model: max zo +
t X
εs+ r
+
r=1
yro zo − n X
j=1 n X
n X
εs− i
(1)
i=1
yrj λj + s+ r = 0 (r = 1, ..., t)
j=1
xij λj + s− i = xio
(i = 1, ..., m)
CR IP T
subject to
m X
(2)
λj = 1
j=1
(j = 1, ..., n) (r = 1, ..., t) (i = 1, ..., m)
AN US
λj ≥ 0 s+ r ≥ 0 s− i ≥ 0
AC
CE
PT
ED
M
where yrj and xij represent the amount of output r (r = 1, . . . , t) produced and the − amount of input i (i = 1, . . . , m) used by museum j, respectively, and zo , s+ r , si and λj are the variables of the optimisation problem. For a detailed introduction to DEA models see e.g. Cooper et al. [16]. The efficiency score Eo is the reciprocal of the optimal value of zo and is comprised between 0 and 1. The value 1 characterises the efficient museums, while the lower the efficiency score, the more inefficient is the museum. For a non efficient museum (with Eo < 1) the value of the variables λj allows you to identify a set of efficient museums that constitutes a “best practice” benchmark. Such a museum could improve its efficiency by simultaneously increasing the value of its outputs while using the same P amount of inputs, or even less. By omitting the constraint nj=1 λj = 1 in the set of constraints (2) and using the same objective function (1), we obtain a model which is characterised by constant returns-to-scale (CRS), named CCR (Charnes-Cooper-Rhodes). The reciprocal of the optimal value of zo in the CRS model gives the CRS efficiency score ECRS,o and can be used to compute a measure of scale efficiency for the museums analysed, given by the ratio between the CRS and VRS efficiency score: SE o =
ECRS,o Eo
(3)
It follows that the global technical efficiency of a museum o (measured by the CRS efficiency score ECRS,o ) is given by the product of the pure technical efficiency score under VRS, Eo , times the scale efficiency factor SE o , that is: ECRS,o = Eo · SE o (see Cooper et al. [16], Basso and Funari [7]). As for the joint implementation of the DEA and BSC approaches, there are a number of studies on profit-oriented organisations, while we are not aware of applications to cultural or more in general nonprofit organisations (with the exception of some applications to the healthcare sector). 11
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
The first seminal attempts to develop models capable of exploiting complementarities between BSC and DEA date back to Rouse et al. [41] and Rickards [39], who published applications to the aircraft maintenance and construction materials, respectively. Other contributions have been published in recent years; for example some papers emphasize the potential of a joint DEA-BSC approach in the measurement of efficiency, proposing the use of a single DEA model with multiple output related to the four traditional perspectives of BSC: Chen and Chen [12] in semiconductor industry, Eilat et al. [19] to evaluate R&D projects in different stages of their lifecycle, Chen et al. [13] for a credit cooperative bank, Chiang and Lin [15] for auto companies and commercial banks. Other papers, instead, choose to work with multiple models and split the comprehensive framework in separated models associated to the various perspectives, even though in different ways: Banker et al. [5] in the US telecommunications industry, Garc´ıa-Valderrama et al. [21] in chemical-pharmaceutical R&D activities, Aryanezhad et al. [4] for a private bank, Amado et al. [2] for the department of equipment maintenance of the Portuguese delegations of a multinational company, Shafiee et al. [43] in the food industry. For example, Banker et al. [5] consider a different performance metrics for each perspective and then use a BCC DEA model with these four performance metrics as outputs. None of these studies, however, take into consideration the museums sector; the present contribution is therefore the first attempt to build a DEA-BSC model specially designed for the evaluation of the museum performance. On the other hand, the idea to measure the performance of museums separately for each BSC perspective may be interesting for museum’s top managers, as will be discussed in Section 4, and the DEA approach facilitates the performance computation. 4. A new two-stage DEA-BSC model for museum performance evaluation
AC
CE
PT
4.1. The model structure In this paper we exploit the capability of DEA to assess the museum performance by integrating the DEA approach with a BSC tool. To this aim we propose a new two-stage DEA-BSC approach. In this approach DEA models are used in the first stage to compute the relative efficiency of museums with respect to the different BSC perspectives and in the second stage to assess the overall relative efficiency. A joint DEA-BSC approach presents some advantages with respect to the use of a pure BSC approach for performance measurement. With regard to this, the benefits of the joint approach are both internal, in relation to the methodology, and external, more operational. The internal advantages regard the synergy between BSC and DEA. On the one hand, BSC allows to reduce the informational overload by focusing the attention on the key success factors, which may suggest the main variables in the DEA model. On the other hand, DEA allows to simultaneously take into account the many input and output factors of BSC, thus permitting to deal with the problem posed by the social purposes of museums, which involves different groups of stakeholders (customers, 12
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
community, professional field; see e.g. the several variables proposed in the literature and summarised in Table 1). In addition, DEA helps to overcome a problem of BSC that arises from the need to determine operational standards (Eilat et al. [19]), by obliging to translate the qualitative concepts expressed in BSC into quantitative variables. The external benefits relate to the reflections of the integrated measurement on decision-making. In the operational management the gap between the good cause of the mission and the list of operational actions to be implemented often raises issues, since the operational actions are not always effectively linked to the mission. With respect to this, the juxtaposition of BSC and DEA allows to guide the transition from the mission to the operations by helping identifying and measuring the objectives of the organisation. Indeed, by comparing different museum organisations, DEA facilitates the identification of suitable benchmarks for the organisation and the dissemination of best practices. Top museum managers have difficulties with the application of incentive mechanisms, due to the risk to worsen the conflicts inside the organisation. The joint DEA-BSC method provides them with more objective performance measures, thus supplying a more evident rationale to support the adoption of incentives. For example, the objective to improve the museum sustainability can be perceived as rather indefinite if it is not accompanied by a measurable indicator. As a matter of fact, the cells of the BSC diagram would require a set of qualitative and quantitative methods in order to be able to measure them completely. Though the DEA model may not be considered exhaustive, it can nevertheless provide an important quantitative component of the assessment of the performance. Let us remember that, in a different sector, Banker et al. [5] proposed a BSC model that measures the performance of each of the four BSC perspectives choosing a single metrics for each perspective (for example the indicator “Return on Assets” for the Financial perspective); then these metrics are considered as the output variables of a BCC DEA model without inputs, equivalent to a CCR model with a constant input. The idea of our contribution is to measure the performance of museums for each BSC perspective by taking into consideration several key indicators instead than just the main one using a DEA model. The performance assessment for all BSC perspectives may be useful because it may help the top management identify the relative efficiency of the museum processes with respect to the different focal points represented by the perspectives. Then, in a second stage, we compute an overall performance measure by applying a BCC DEA model with the DEA performance scores of the four BSC perspectives as outputs and a constant input; for a graphical representation see Figure 1. The choice of BCC models is natural, since it allows the presence of variable returns-to-scale encompassing at the same time, as a special case, the presence of constant returns-to-scale. As for the model orientation, we assume that a museum generally try to use the resources at hand in order to optimise the fulfillment of the museum mission. Under this assumption, an output orientation seems a natural choice. 13
ACCEPTED MANUSCRIPT
Internal process perspective
Innovation and learning perspective
Overall performa nce measure
Financial perspective
CR IP T
Customer perspective
Figure 1: Basic flow chart of the two-stage DEA-BSC model. The four squared boxes represent the DEA models defined in the first stage for the BSC perspectives; the round box depicts the DEA model of the second stage, which exploits the four performance scores obtained in the first stage in order to compute a unique overall performance measure.
AC
CE
PT
ED
M
AN US
Notice that we do not implement a direct hierarchy among the DEA models that measure the performance along the four BSC perspectives. This allows the model to be quite simple and easily understandable by the museum’s top managers. On the other hand, we have remarked in Section 2 that in the cultural sector the hierarchy among the BSC perspectives is not clearly defined, so much so that various scholars interpret the hierarchy in different ways or do not impose a hierarchy at all. At the same time, the separate measurement of the performance with respect to the four BSC perspectives is valued by the museum’s top managers since it may help to highlight the strengths and weaknesses of the organisation. Ultimately, we propose a two-stage DEA-BSC model organised as follows: in the first stage we identify the variables relevant for museum organisations and analyse the performance along each perspective suggested by BSC with a different DEA model, while the second stage synthesises the performance indicators of the various perspectives in an overall performance measure computed with an additional DEA model. As for the choice of the variables of the DEA models associated to the four BSC perspectives, we may observe from Table 1 that the papers on BSC for museums (Haldma and L¨a¨ats [22], Marcon [32], Zorloni [50]) list a number of different indicators; some of them, however, can only be obtained by carrying out ad hoc surveys or require the access to detailed internal data, and actually only Haldma and L¨a¨ats [22] presents an empirical application to a museum. Since our aim is to compare the performance of a set of museums, we need to choose indicators that are at the same time focused on the main characteristics of the museum activities and computable for all the museums analysed. The input (I) and output (O) variables chosen for each model are the following (see also Figure 2). Customer perspective I: insured value; it is used as a proxy for the value of the museum exhibits 14
ACCEPTED MANUSCRIPT
Input:
Outputs:
Insured value
1. Visitors 2. Web site visits 3. Members 4. Donations 5. Catalogues
CR IP T
CUSTOMER PERSPECTIVE MODEL
Input:
Outputs:
Total costs
1. Conservation, restauration costs 2. Amount spent for new acqusitions 3. Visitors
AN US
INTERNAL PROCESS PERSPECTIVE MODEL
SECOND STAGE MODEL
Outputs:
Input:
1. Customer perspective score
Constant
2. Internal process perspective score 3. Innovation and learning perspective score
Outputs:
Constant
1. Personnel training 2. Sustainability indicators: 2a. Innovative lighting 2b: Environmental sustainability 2c: Facilities for people with disabilities
PT
ED
Input:
4. Financial perspective score
M
INNOVATION AND LEARNING PERSPECTIVE MODEL
FINANCIAL PERSPECTIVE MODEL Input:
Outputs:
1. Income from tickets 2.Sponsorships, donations, public funding 3. Other incomes
AC
CE
Expenditure
Figure 2: Flow chart of the two-stage DEA-BSC model. The four boxes on the left side illustrate the input and output variables of the DEA-BSC models of the first stage; the box on the right displays the input and output variables of the DEA model of the second stage.
15
ACCEPTED MANUSCRIPT
O: number of visitors, web site visits, members, catalogues, (value of) donations Notice that the output variables in the Customer perspective relate to the stakeholders of the museum. Internal process perspective I: total costs
CR IP T
O: conservation and restoration costs, amount spent for new acquisitions, number of visitors
Innovation and learning perspective I: constant
AN US
In the Internal process perspective the output variables are related to the main purposes of the museum, namely the preservation of cultural heritage, its enhancement through an expansion of museum’s collections, and the increase in value for the museum’s customers.
O: personnel training (cost or number of hours per employee), an indicator of innovations aiming at improving the enjoyment of exhibits, an environmental indicator and an indicator of museum accessibility
CE
PT
ED
M
The outputs in the Innovation and learning perspective measure the efforts undertaken to continuously increase the skills of the museum personnel, on the one hand, and three different facets of innovation in a museum context. A more detailed discussion on how to define the innovation indicators is presented in Subsection 4.2, but they will be defined as a measure of the degree of innovation and sustainability of the museum and will not depend on the museum size. By analogy, the personnel training indicator is computed as either the cost or the number of hours per employee, hence regardless of the museum size. For the Innovation and learning perspective model, therefore, it is not necessary to consider an input variable; this situation corresponds to the case of a model with a single constant input. Financial perspective
AC
I: expenditure
O: income from tickets, sponsorships donations and public funding, other incomes
In the Financial perspective the outputs are given by museum’s income, divided into its main items. The results of the first stage models can be effectively summarised by means of an interesting geometric representation of the performance scores obtained for the four perspectives: a radar chart in which we represent along each of the four semiaxes the score of a different perspective; an example is shown in Figure 3. What we obtain is a diamond; the biggest diamond is related to the case of a museum which 16
ACCEPTED MANUSCRIPT
Customer 1.0 0.8 0.6 0.4 0.2 0.0
Innovation
Int.process
CR IP T
Financial
AN US
Figure 3: Radar chart showing the score of a different perspective along each of the four semiaxes; the values of the performance scores shown are equal to 1 for the Customer perspective, 0.5 for the Internal process perspective, 0.8 for the Innovation and learning perspective, 0.2 for the Financial perspective.
is fully efficient with respect to all the four perspectives, i.e. with efficiency scores all equal to 1, while the diamond “shrinks” asymmetrically when the efficiency score of a perspective decreases.
AC
CE
PT
ED
M
The DEA-BSC models of the first stage can be appreciated by museum’s management bodies for the detailed information provided about the performance of BSC perspectives. On the other hand, it may be useful, and certainly it arouses interest, to compute a single measure of the (relative) overall performance of the organisation. Think, for example, of the degree mark that synthesises the marks obtained in all the courses of an academic degree program: it may not be considered as essential, but it is commonly provided as a useful information. Hence, in the second stage we define a further DEA model which combines the efficiency scores obtained for the various BSC perspectives into an overall performance indicator. As is well known, the DEA approach provides a relative measure of performance, so at least one of the museums will turn out to be efficient and will receive an overall score of 1. This does not necessarily require that the efficient museums are efficient also with respect to all the four DEA-BSC models of the first stage, hence for all BSC perspectives (of course they will have to be efficient for at least some of the perspectives). Second stage model I: constant O: the efficiency scores of the DEA models for the four perspectives: Customer, Internal process, Innovation and learning and Financial perspective scores. As pointed out, two of our DEA models present a single constant input, a situation that has become less uncommon in recent years Karagiannis and Lovell [28]: both 17
ACCEPTED MANUSCRIPT
AN US
CR IP T
the Innovation and learning perspective model and the second stage model, in which the output variables are not proportional to the museum size. This feature has some interesting consequences from the point of view of the properties of the resulting DEA model. Indeed, Lovell and Pastor [30] proved that an output oriented BCC model with a single constant input, as is the case for our models, is equivalent to an output oriented BCC model without inputs, hence a “pure output” model. In addition, Lovell and Pastor [30] proved that an output oriented CCR model with a single constant input coincides with the corresponding BCC model with the same variables. Actually, an output oriented BCC model with a single constant input is equivalent to the corresponding PnCCR model, since in the case of a single constant input the convexity constraint j=1 λj = 1 is redundant. As a consequence, in the case of a single constant input the scale efficiency is equal to 1 and the returns to scale are globally constant. Note that the concept of a constant input is analogous to that of a “dummy input” equal to one for all decision-making units adopted in Cherchye et al. [14].
ED
M
4.2. Sustainability indicators for the museum activities In recent years sustainability issues have aroused more and more attention in the museum sector too; see for example [36] and Ernst et al. [20]. The efforts made in order to make the museum more sustainable, therefore, deserve to be properly taken into account when measuring the museum performance. Nevertheless, to our knowledge, scholarly literature has not yet come to propose synthetic indicators to measure museum sustainability. In this paper, as anticipated, in order to investigate whether museums are innovative and responsible for environmental impact, and whether they pay attention to social problems, we identify three groups of items:
PT
1. use of innovative lighting, 2. environmental sustainability, 3. presence of facilities for people with disability
AC
CE
and measure them by means of three numerical indices. With regard to the innovative lighting, we focus on the LED lights and measure this aspect with a binary variable that takes either value 0 (denoting the absence of LED lighting) or 1 (denoting the presence of LED lighting). In order to measure the second and third issues, we conducted some interviews with managers of the Venice Municipal Museums Foundation; as a result, we detail aspects 2 and 3 as follows: • Environmental sustainability: 1.use of energy-saving lamps; 2.presence of high efficiency systems; 3.use of water saving devices; 4.differentiated waste collection; 5.use of eco-friendly products (in the daily cleaning services, construction and restoration works and office material); 6.thermal building insulation (either total or partial); 7.monitoring of the museum’s energy consumption; 8.procurement of power supply. 18
ACCEPTED MANUSCRIPT
• Facilities for people with disability: 1. presence of lifts; 2. presence of ramps.
AN US
CR IP T
By counting the number of items present (assigning 0 when an item is absent and 1 when it is present and then summing up), we obtain an index for environmental sustainability whose values are comprised between 0 and 8 (the overall number of green items considered). Similarly, we can define an index associated to the facilities for people with disability, with values between 0 and 2. We also define a comprehensive indicator of innovation and sustainability, called sustainability indicator for the sake of brevity and computed as the weighted sum of the three indices associated to the innovative lighting, environmental sustainability and facilities for people with disability. If all items can be considered equally important (with weights equal to 1), the comprehensive indicator is simply the sum of the values of the three indices and takes values between 0 and 11, where the maximum value is reached by the museums that fulfill all items. This comprehensive indicator will be used in the empirical analysis of the Venetian museums, since the differences in some of the innovation and sustainability items were limited for the museums analysed. 5. Restricting virtual outputs weights
AC
CE
PT
ED
M
In Subsection 4.1 we have proposed a two-stage DEA-BSC model for the evaluation of the performance of museums. As usual for standard DEA models, in the computation of the performance score this model emphasises the input and output variables according to which the museum best performs. This essential feature of DEA models, however, may sometimes lead to a lack in discriminatory power, especially when we have several variables and few organisations. Actually, it may well happen that several museums are assessed as efficient, and this may regard a considerable percentage of the organisations analysed if their number is low. Moreover, the DEA efficiency may even be achieved by disregarding some important variables in the optimization process, if their weight is negligible in the optimal solution. For instance, a museum could be stated as efficient even though the number of visitors is minimal, thus neglecting an important aspect related to its mission. A way to tackle both issues at the same time is to resort to restrictions on the weights associated to the variables of the DEA program. These weights restrictions, on the other hand, allow to incorporate some value judgments that express the decision maker’s preferences in a DEA model. In the DEA literature we find various types of weights restrictions, usually grouped into four broad categories (even if we may find in the literature also a few different approaches, see e.g Rogge [40]): • absolute weights restrictions; • assurance regions of type I, that impose restrictions on the ratio between the weights of either the inputs or the outputs; • assurance regions of type II, that impose cross restrictions on the relationship between the input and output weights; 19
ACCEPTED MANUSCRIPT
• proportional restrictions on virtual inputs and outputs.
ur 0 y r 0 j ≤ Ur0 Lr0 ≤ Pt u y r rj r=1
(r0 = 1, . . . , t;
CR IP T
For a review see for example Allen et al. [1], Angulo-Meza and Lins [3] and Thanassoulis et al. [46]. With regard to our two-stage DEA-BSC model for the evaluation of the performance of museums we have decided to adopt the fourth approach, setting restrictions on virtual outputs. Proportional restrictions on virtual outputs set lower and upper limits on the proportion of each virtual output (defined as the product of the level of output r0 and the related optimal weight) with respect to the total virtual output: j = 1, . . . , n)
(4)
(i0 = 1, . . . , m;
M
vi0 xi0 j ≤ Ui0 L i 0 ≤ Pm i=1 vi xij
AN US
where Lr0 and Ur0 represent the lower and upper percentage values set for the virtual output r0 ; for example, we may require that the proportion of the total virtual output for museum j due to output r0 (say the number of visitors) is comprised between 40 % and 90 %. Virtual weights restrictions were originally proposed by Wong and Beasley [49]; for an in-depth analysis see also Sarrico and Dyson [42]. Analogously, proportional restrictions may be defined also for virtual inputs: j = 1, . . . , n)
(5)
AC
CE
PT
ED
In our model we do not apply restrictions on the virtual inputs because the DEA models considered are all based on a single input and therefore it does not make sense to consider proportional restrictions on inputs. The virtual weights restrictions defined in constraints (4) (and (5)) have the advantage to consider the relative contribution of each output (and input) to the total virtual output (and input). As such, it may be more intuitive and thus more easily understood by decision makers (see Sarrico and Dyson [42]). This is an important advantage in the museum sector, since the museum managers do not often have an operational research background. Moreover, the virtual weights restrictions allow to take into account value judgements of decision makers in a more straightforward way, and this is especially useful in the cultural sector, where we do not have a production technology that gives rise to evident technological constraints like in Podinovski [38]. In order to present the DEA-BSC models with weights restrictions on the virtual outputs, let us adopt a two-phase approach (on the two-phase solution method see for example Cooper et al. [16]). As discussed in the literature on proportional weights restrictions (Wong and Beasley [49] and Sarrico and Dyson [42]), such restrictions may be applied in two different ways. Actually, when we formulate the optimisation problem that allows to compute the efficiency score of museum o, we may either consider restrictions (4) for all the museums to be evaluated, thus adding t × n constraints to the optimisation problem, or reduce the number of weights constraints considered and add only the t constraints related to the target museum o while disregarding the others. 20
ACCEPTED MANUSCRIPT
Let us consider the first case, which corresponds to the model chosen for our analysis. Each of the four models associated to the four perspectives in the first stage, as well as the second stage model, have a similar DEA structure and may be formalised in primal form (the so called multipliers form) as follows: min vxo − vo
t X
(7)
ur yrj − vo ≥ 0
(j = 1, . . . , n)
ur0 yr0 j ≤ Ur0 L r 0 ≤ Pt r=1 ur yrj v ≥ 0, ur ≥ 0
(r0 = 1, . . . , t;
r=1
or equivalently:
(8)
j = 1, . . . , n)
AN US
vxj −
ur yro = 1
r=1 t X
CR IP T
subject to
(6)
(r = 1, . . . , t)
min vxo − vo t X
r=1
r=1
ur (Ur0 yrj ) − ur0 yr0 j ≥ 0 v ≥ 0,
ur ≥ 0
(10) (11)
(12) (j = 1, . . . , n)
ur (Lr0 yrj ) + ur0 yr0 j ≥ 0
CE
t X
AC
r=1
ur yrj − vo ≥ 0
PT
−
t X
ur yro = 1
r=1 t X
ED
vxj −
M
subject to
(9)
(13)
(r0 = 1, . . . , t;
j = 1, . . . , n)
(14)
(r0 = 1, . . . , t;
j = 1, . . . , n)
(15)
(r = 1, . . . , t)
(16)
Notice that the term −vo in the Pnobjective function (11) is associated to the variable returns-to-scale constraint j=1 λj = 1 present in the dual of the primal program (11)–(16). In the case of a model with constant returns-to-scale, instead, this term is omitted. In order to express the linear program (11)–(16) in matrix form and write the associated dual program, let us first equivalently write constraints (14) as follows: uQLj ≥ 0 (j = 1, . . . , n) 21
(17)
ACCEPTED MANUSCRIPT
where u = (u1 , . . . , ut ) is the row vector of the output weights, 0 ∈ Rt is a null row vector and QLj is the following t × t matrix: (−L1 + 1)y1j −L2 y1j ... −Lt y1j −L1 y2j (−L2 + 1)y2j . . . −Lt y2j QLj = (18) = Dj − yj L .. . −L1 ytj −L2 ytj . . . (−Lt + 1)ytj
AN US
CR IP T
Here L = (L1 , . . . , Lt ) is the row vector of the lower bounds on the proportional virtual outputs, yj = (y1j , . . . , ytj )0 is the j-th column of the t × n matrix Y of the output variables (yj is the output vector of the j-th museum) and Dj is the diagonal matrix with the elements of yj on the main diagonal. Let us observe that constraints (14) are thus written as a set of t constraints (one for each output) for each museum j (j = 1, . . . , n). By defining the block matrix QL = QL1 QL2 . . . QLn (19) the whole set of constraints (14) can be written in compact form as follows: uQL ≥ 0
(20)
Notice that QL has dimension t × (tn) and 0 is the null row vector with tn elements. Analogously, constraints (15) can be equivalently written as follows:
M
uQUj ≥ 0 (j = 1, . . . , n)
PT
ED
where QUj is the t × t matrix: (U1 − 1)y1j U2 y1j ... Ut y1j U1 y2j (U2 − 1)y2j . . . Ut y2j QUj = .. . U1 ytj U2 ytj . . . (Ut − 1)ytj
= −Dj + yj U
(21)
(22)
AC
CE
and U = (U1 , . . . , Ut ) is the row vector of the upper bounds on the proportional virtual outputs. In compact form we may define the block matrix QU = QU1 QU2 . . . QUn (23) and write the whole set of constraints (15) as follows: uQU ≥ 0
(24)
Finally, by letting X be the m × n matrix of the input variables, xo be the o-th column of X (note that in our case m = 1 since we have only 1 input variable and xo is the input of the target unit o) and 1 = (1, . . . , 1) ∈ Rt , the primal program in (multiplier) matrix form can be written as follows: min vxo − vo 22
(25)
ACCEPTED MANUSCRIPT
subject to uyo = 1 vX − uY − vo 1 ≥ 0 uQL ≥ 0 uQU ≥ 0 v ≥ 0, u ≥ 0
(26) (27) (28) (29) (30)
subject to
AN US
uyo = 1 vX − uY − vo 1 ≥ 0 uQLo ≥ 0 uQUo ≥ 0 v ≥ 0, u ≥ 0
CR IP T
Let us observe that from the linear program (25)–(30) it is immediate to deduce the primal program for the case in which the virtual weights constraints are imposed only to the target museum: min vxo − vo (31) (32) (33) (34) (35) (36)
M
We can now write the dual of the primal program (25)–(30), i.e. the model in envelopment form: max zo (37) subject to
(38) (39) (40) (41)
max s− + 1s+
(42)
Xλ + s− = xo zo∗ yo − Y λ + QL τ L + QU τ U + s+ = 0 1λ = 1 L λ ≥ 0, τ ≥ 0, τ U ≥ 0, s− ≥ 0, s+ ≥ 0
(43) (44) (45) (46)
PT
ED
Xλ ≤ xo zo yo − Y λ + QL τ L + QU τ U ≤ 0 1λ = 1 λ ≥ 0, τ L ≥ 0, τ U ≥ 0
AC
CE
where zo is the dual variable associated to constraint (26), λ ∈ Rn is the column vector of the dual variables associated to constraints (27), τ L ∈ Rtn and τ U ∈ Rtn are the column vectors of the dual variables associated to the lower and upper virtual weights constraints (28) and (29), respectively. The optimal solution of the primal and dual programs (25)–(30) and (37)–(41), denoted by zo∗ , provides the reciprocal of the efficiency score of museum o. In phase II, by setting zo = zo∗ and solving the following linear program, the sum + of the input and output slacks s− ∈ R and s+ = (s+ 1 , . . . , st ) is maximised: subject to
23
ACCEPTED MANUSCRIPT
We can now formally define the virtual weights restriction efficiency (VWRefficiency) and the projection on the VWR efficient frontier (VWR-projection) of the inefficient museums.
CR IP T
Definition 1. Let (zo∗ , λ∗ , τ L∗ , τ U ∗ , s−∗ , s+∗ ) be an optimal solution of the Phase I dual program (37)–(41) and Phase II program (42)–(46), with o ∈ { 1, . . . , n }. Museum o is called VWR-efficient if and only if this solution satisfies zo∗ = 1, s−∗ = 0 and s+∗ = 0, i.e. if it has efficiency score equal to 1 and is zero-slack. Definition 2. Let (zo∗ , λ∗ , τ L∗ , τ U ∗ , s−∗ , s+∗ ) be an optimal solution of the Phase I dual program (37)–(41) and Phase II program (42)–(46), with o ∈ { 1, . . . , n }. The VWR-projection on the efficient frontier of an inefficient museum o, (xˆo , yˆo ), is represented by:
AN US
xˆo = xo − s−∗ = Xλ∗ yˆo = zo∗ yo + QL τ L∗ + QU τ U ∗ + s+∗ = Y λ∗
(47) (48)
PT
ED
M
Similar definitions may be formulated also for the case in which the virtual weights constraints are applied only to the target museum. Nevertheless, it is worth pointing out that in this case the peer group of an inefficient museum o, i.e. the set of museums with λ∗j > 0, may include also museums that are not efficient due to the weights constraints. Moreover, in such a case the VWR-projection may not satisfy the virtual weights constraints, since it is computed as a linear combination of museums for which such constraints are not imposed. Hence the suggestion that the model in which the virtual weights restrictions are applied to all museums simultaneously may be preferable. Actually, this is the model that will be primarily used in the empirical application that will be presented in Section 6, even if in this case the lower and upper bounds have to be carefully chosen in order to ensure that the feasible region is non empty.
CE
6. An empirical analysis on the performance of the MUVE museums with the DEA-BSC model
AC
6.1. The data In this section we present an empirical application of the two-stage DEA-BSC model proposed in a real context. In this application we evaluate the relative performance of the 11 municipal museums of Venice, whose organisation, development and promotion have been entrusted to the MUVE Foundation1 , the most important Venetian museum foundation. This allows us to compare the performance of museums with similar management policies. The data on the municipal museums of Venice have been kindly provided by the MUVE Foundation and refer to the year 2013 (see Tables 3 and 4). 1
http://www.visitmuve.it/en/home/
24
ACCEPTED MANUSCRIPT
Table 3: MUVE museums: non financial data (percentage values).
29.69 16.62 4.90 14.93 9.07 9.88 4.59 1.26 0.56 1.09 7.41 100.00
Number of members 20.22 33.71 1.12 12.13 8.31 11.24 6.97 5.17 0.00 0.00 1.12 100.00
Catalogues sold 18.44 1.78 39.25 4.34 3.65 18.24 0.20 3.16 0.00 8.88 2.07 100.00
Number of web visits 50.80 12.45 5.39 5.48 5.89 5.62 3.52 1.46 1.33 1.07 7.00 100.00
Insured value
Personnel training indicator 9.96 11.48 8.78 9.48 10.45 7.44 8.44 9.60 9.67 7.47 7.24 100.00
Sustainability
57.73 12.08 5.13 7.62 4.67 4.88 1.95 0.82 0.47 0.96 3.70 100.00
ED
M
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History Museum Mocenigo Palace Carlo Goldoni’s House Clock Tower Lace Museum Fortuny Palace Total
Number of visitors 65.64 11.73 6.61 4.95 3.03 3.11 0.41 0.86 0.40 1.29 1.96 100.00
CR IP T
Total area
AN US
Museums
Table 4: MUVE museums: financial data (percentage values).
CE
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace Total
AC
Income from tickets 79.11 7.01 4.33 3.03 1.91 1.37 0.19 0.36 0.38 0.54 1.77 100.00
PT
Museums
Public funding
Sponsorships
Donations
Other incomes
Expenditure
47.79 0.00 3.82 0.00 0.00 48.39 0.00 0.00 0.00 0.00 0.00 100.00
2.69 36.10 0.00 2.98 2.89 2.63 44.28 0.00 7.09 1.34 0.00 100.00
98.04 0.00 1.08 0.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
78.22 11.32 1.19 2.73 1.75 1.93 0.63 0.13 0.13 0.15 1.81 100.00
30.95 19.31 4.01 10.70 10.71 9.06 4.34 2.77 0.55 1.69 5.92 100.00
25
Conservation costs 8.59 26.73 1.44 23.53 14.72 0.59 0.00 2.46 18.74 0.00 3.21 100.00
Personnel costs 25.55 18.93 5.72 12.97 11.75 11.38 2.99 4.01 0.53 2.04 4.12 100.00
10.23 10.23 10.23 10.23 10.23 10.23 5.68 10.23 4.55 9.09 9.09 100.00
ACCEPTED MANUSCRIPT
Table 5: MUVE museums: sustainability indicators. Innovative lighting
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1 1 1 1 1 1 1 1 0 1 1
Environmental sustainability 6 6 6 6 6 6 4 6 4 6 5
Facilities for people with disability 2 2 2 2 2 2 0 2 0 1 2
Sustainability indicator 9 9 9 9 9 9 5 9 4 8 8
CR IP T
Museums
AC
CE
PT
ED
M
AN US
It can immediately be noted the presence of a museum which is much bigger than the others. It is the Doge’s Palace, which is located in S. Mark’s square and represents a sort of “symbol of Venice”. This is by large the biggest museum of the MUVE Foundation: it attracts the majority of visitors and web visits, draws most of the donations and exhibits the highest percentages of expenditure and income earned. On the other hand, other museums are able to capture the highest number of members (Correr Museum) and to sell the highest number of catalogues (Glass Museum). Among the output variables of the model for the Internal process perspective we find the conservation and restoration costs and the amount spent for new acquisitions. With respect to the conservation and restoration costs, Correr Museum and Ca’ Rezzonico prevail. The cost for new acquisitions in the year considered, instead, is null for all the municipal museums. We gathered also information on the three groups of issues related to the presence of innovative lighting, environmental sustainability, and facilities for people with disability; in this respect, we have computed both the three relative indicators and the sustainability indicator presented in Subsection 4.2. In the empirical analysis we will consider the comprehensive sustainability indicator given the limited differences observed for the MUVE museums. From Table 5, which shows the values of the sustainability indicators for the museums analysed, we may observe that the municipal museums of Venice on the whole pay much attention to the issues related to innovation and sustainability. The last column of Table 5 displays the value of the comprehensive sustainability indicator; it is interesting to note that a good 7 museums out of 11 reach a level of 9 (with a maximum value for the indicator equal to 11), while the Mocenigo Palace and Clock Tower have wide margins for improvement with regard to sustainability. 6.2. First and second stage performance analysis In this subsection we present the results obtained by applying the two-stage DEA-BSC model with virtual weights restrictions proposed in Sections 4 and 5. 26
ACCEPTED MANUSCRIPT
Table 6: Virtual weights restrictions applied in the analysis of the MUVE museums.
LC 1 LC 2 LC 3 LC 4 LC 5
= = = = =
LIP 1 = LIP 2 =
0.4 0 0 0 0
Upper bound Ur
0 0.05
LI1 = LI2 =
0.3 0.2
LF 1 = LF 2 = LF 3 =
0.4 0 0
L21 L22 L23 L24
0.2 0.05 0.05 0.05
M
= = = =
U1C U2C U3C U4C U5C
= = = = =
0.9 0.5 0.5 0.5 0.5
CR IP T
Lower bound Lr
AN US
Virtual weight output Stage 1 Customer model Visitors Web site visits Members Donations Catalogues Internal process model Conservation, restoration costs Visitors Innovation and learning model Personnel training Sustainability indicator Financial model Income from tickets Sponsorships, donations, public funding Other incomes Stage 2 Second stage model R Customer performance score EC R Internal process performance score EIP Innovation and learning performance score EIR Financial performance score EFR
U1IP = U2IP =
0.95 1
U1I = U2I =
0.8 0.7
U1F = U2F = U3F =
0.9 0.6 0.6
U12 U22 U32 U42
0.8 0.5 0.5 0.5
= = = =
AC
CE
PT
ED
The lower and upper bounds defining the virtual weights restrictions are reported in Table 6. The choice of these bounds has been made starting from some suggestions arisen from a number of interviews to managers of the MUVE Foundation, useful to determine the relative importance of the variables. Notice that the restrictions set in Table 6 take into account the peculiarities of the context in which the museums analysed operate. For example, in the European context, donations are not so relevant as they are in the US, and this accounts for the choice to set a lower range for the value of the weight associated to this variable. On the other hand, visitors cannot be set aside given their importance in the museum mission, so the restriction on the weight associated to the number of visitors ensures that it will obtain a high value. A similar reasoning affects also the weights restrictions in the second stage model: in this model none of the perspectives can be neglected, but the importance of the performance score of the Customer perspective is emphasised with respect to the others. Moreover, we point out that the widths of the virtual weights intervals depend also on the need to have a non empty feasible region for the optimisation problems presented in Section 5, which prevents the intervals from being too narrow. Table 7 summarises the overall results of the two-stage DEA-BSC model with virtual weights restrictions. Columns 2 to 5 of this table report for each perspective the value of the performance scores computed by using the first stage models for 27
ACCEPTED MANUSCRIPT
Table 7: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions for all museums.
Customer persp.
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1.0000 0.9064 1.0000 0.5797 0.5959 0.6136 0.3589 1.0000 1.0000 1.0000 0.5570
Internal process persp. 1.0000 0.4952 0.8426 0.5120 0.3180 0.1668 0.0476 0.1666 1.0000 0.4512 0.1736
Innovation and learning persp. 0.9472 1.0000 0.9062 0.9305 0.9642 0.8600 0.6775 0.9348 0.7145 0.7939 0.7860
Financial persp. 1.0000 0.1711 0.3695 0.1110 0.0702 0.0691 0.0334 0.0493 1.0000 0.1321 0.1256
1.0000 0.5283 0.7155 0.4278 0.3743 0.3293 0.2053 0.4276 0.9840 0.5081 0.3311
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
AC
CE
PT
ED
M
the four BSC perspectives; the last column exhibits the overall performance score obtained with the second stage model. As it can be expected, the relative performance scores obtained by a museum for the different BSC perspectives may differ. In general, we may note that the number of efficient museums is relatively high for the Customer perspective (5 out of 11) while we have only 1 or 2 efficient museums for the other perspectives. This may be due to the fact that the DEA model for the Customer perspective has the highest number of output variables. On the whole, it seems that the insertion of the virtual weight constraints in the DEA models is effective in keeping the number of museums that are assessed as efficient low. It is also interesting to note that no museum is efficient with regard to all perspectives: no one is perfect. More in detail, two museums, Doge’s Palace and Clock Tower, are efficient as to the Customer, Internal process and Financial perspectives, and four other museums are efficient only as to one perspective (the Correr Museum as to the Innovation perspective, the Glass, Carlo Goldoni’s House and the Lace Museums as to the Customer one). The remaining museums do not achieve efficiency as to any perspective. The results obtained with the four DEA-BSC models at the first stage may be conveniently displayed using a radar chart as suggested by Amado et al. [2] (see Graph 1, p. 398) and are reported in Figure 4. In addition, we have also introduced in Subsection 4.1 a geometrical representation of the performance scores obtained for the four perspectives by a museum using a radar chart with the scores along the four axes (see Figure 4). Figure 5 shows the radar charts obtained for all the MUVE museums. By comparing the values of the performance scores achieved by a museum along the different perspectives we may identify its strengths and weaknesses with respect to the best performers. On the whole, the MUVE museums show better performances 28
ACCEPTED MANUSCRIPT
DP 1.0
FP
CM
0.8 0.6
LM
GM
0.4
Customer perspective 0.2 Internal process perspective 0.0
CT
CR
Innovation perspective
CGH
CP MP
NHM
CR IP T
Financial perspective
AN US
Figure 4: Efficiency scores obtained by all museums for the different perspectives in the first stage. Legend: DP:Doge’s Palace, CM:Correr Museum, GM:Glass Museum, CR:Ca’ Rezzonico, CP:Ca’ Pesaro, NHM:Natural History Museum, MP:Mocenigo Palace, CGH:Carlo Goldoni’s House, CT:Clock Tower, LM:Lace Museum, FP:Fortuny Palace.
AC
CE
PT
ED
M
as to the Customer perspective and the Innovation and learning perspective, while several museums exhibit a low performance score as to the Financial and the Internal process perspectives. In a nutshell, the MUVE museums strengths seem to be their ability to attract visitors and to arouse interest and passion, on the one hand, and the ability to train employees and to undertake sustainability actions, on the other hand. Their weaknesses, instead, seem to concern the conservation and acquisition processes as well as the fund raising operations. Moreover, we notice that the museum that is closer to the complete efficiency at the first stage is the Doge’s Palace, by large the MUVE biggest and most visited museum; and actually this is the only museum that is efficient in the second stage model which provides the overall performance indicator. As for the results obtained for the overall performance indicator in the second stage model (last column of table 7), we can note the dispersion of the performance scores. More in detail, we may distinguish 3 sets of museums: Mocenigo Palace, Natural History Museum, Fortuny Palace and Ca’ Pesaro exhibit the lowest performance scores (comprised between 0.20 and 0.37); Carlo Goldoni’s House, Ca’ Rezzonico, Lace, Correr and Glass Museums obtain an intermediate score (in the range 0.42 to 0.71), while the performance score of the Clock Tower, near 1, sets this museum close to the most efficient one (Doge’s Palace). Therefore, the DEA-BSC model with virtual weights restrictions turns out to be rather strict with the museums exhibiting an inefficiency as to one or more BSC perspectives. It is known that the DEA approach allows to suggest to the non efficient units a benchmark, given by the so called reference set, that can be looked at for improvements. From the point of view of museums, the reference set can be considered in a 29
ACCEPTED MANUSCRIPT
Correr Museum
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Financial
Innovation
Ca' Rezzonico Customer
ED
Financial
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Natural History Museum Customer
Financial
Innovation
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Innovation
PT CE AC
Financial
Int.process
Innovation
M
Customer
Financial
Int.process
Innovation
Ca' Pesaro
1.0 0.8 0.6 0.4 0.2 0.0
AN US
Customer
Financial
Int.process
Innovation
Glass Museum 1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
CR IP T
Financial
Doge’s Palace
Mocenigo Palace
Carlo Goldoni’s House
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Financial
Int.process
1.0 0.8 0.6 0.4 0.2 0.0
Innovation
Innovation
30
Int.process
ACCEPTED MANUSCRIPT
Financial
Clock Tower
Lace Museum
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Financial
Innovation
CR IP T
Customer
Financial
Int.process
Innovation
Fortuny Palace 1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
AN US
Innovation
Figure 5: DEA-BSC performance scores obtained for the various museums at the first stage; each radar chart shows along each of the four semiaxes the score of a different perspective.
AC
CE
PT
ED
M
sense as a benchmark utilising the best practices, to be imitated when possible. Table 8 reports the reference set of the various museums obtained with the DEA-BSC models in the first stage for the different four perspectives. The reference set of the second stage is not reported since it consists of the only efficient museum, Doge’s Palace, for all museums. As for the Internal Process and the Financial perspectives, the reference set of all inefficient museums is composed of the same pair of efficient units (Doge’s Palace and the Clock Tower), the only efficient museums. The situation is more diversified, instead, for the Customer perspective; in this case, the reference set is still composed of two efficient museums, but the pairs are not always the same. It is interesting to note that the Glass Museum enters the reference set of six other museums, while on the contrary the Clock Tower is not present in the reference set of any other. This means that six museums can draw inspiration from the Glass Museum – in combination with another efficient museum – with respect to the marketing policies related to the Customer perspective. In the end, the outcomes of the DEA-BSC model are not limited to the detection of the efficient museums, that may be used to define proper incentive mechanisms for the museums’ managers. The computation of the inefficiency scores for the non efficient museums is especially useful from the point of view of the policy and managerial implications, since it may be used to identify the weaknesses of the museum organisations. For instance, let us consider the performance score of the Mocenigo Palace for the Internal process perspective (Table 7). This is the lowest score of all MUVE museums with respect to this perspective, and this is indicative of the fact that this museum fails to reach the goals of the Internal process. By looking more closely at the data, 31
ACCEPTED MANUSCRIPT
Table 8: Reference sets of the various museums obtained with the DEA-BSC models of the first stage for the different four perspectives. Legend: DP:Doge’s Palace, CM:Correr Museum, GM:Glass Museum, CR:Ca’ Rezzonico, CP:Ca’ Pesaro, NHM:Natural History Museum, MP:Mocenigo Palace, CGH:Carlo Goldoni’s House, CT:Clock Tower, LM:Lace Museum, FP:Fortuny Palace.
{DP} {DP, GM} {GM} {DP, GM} {GM, LM} {GM, CGH} {GM, LM} {CGH} {CT} {LM} {GM, LM}
Innovation persp.
Financial persp.
{CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM}
{DP} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {CT} {DP, CT} {DP, CT}
CR IP T
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Nat. Hist. M. Mocenigo Palace C. Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
Internal process persp. {DP} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {CT} {DG, CT} {DG, CT}
AN US
Customer persp.
ED
M
we may see that this museum exhibits at the same time relatively high total costs and few visitors, and in the year considered it does not fulfill the conservation and restoration objective in any way. Hence, an improvement is certainly advisable for this museum as to the Internal process perspective. Since the Mocenigo Palace is a museum of textiles and costumes, new well promoted temporary exhibitions centered on historical Venetian textiles might be able to arouse the interest of foreign tourists visiting Venice and to attract a higher number of visitors.
AC
CE
PT
6.3. Changing the model options We have seen that the results obtained for the DEA-BSC model with virtual weights restrictions indicate that the museums that are inefficient with respect to one o more BSC perspectives are rather severly penalised in the overall performance indicator of stage 2. In order to see if this penalisation is worsened by the presence of the bounds in the model, we have investigated the effect of the inclusion of the upper and lower bounds in the second stage model. More precisely, we have compared the overall performance scores obtained in the second stage considering the restrictions reported in Table 6 to the results calculated with the analogous model without any second stage restriction and with the analogous model including only the upper bounds. The comparison of the results are summarised in Figure 6. As can be see, the results obtained with and without bounds are rather different. In detail, if we compare the performance scores of the unrestricted model to those of the model with upper bounds, it is clear that the introduction of the upper bounds does affect the results. Moreover, when we include also the lower bounds, the performance scores further sensibly decreases. Therefore the bounds imposed on the virtual weights in the DEA model, that neither allow to neglect the result obtained 32
ACCEPTED MANUSCRIPT
Doge’s Palace Correr Museum Glass Museum Ca' Rezzonico Ca' Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Fortuny Palace 0
0.2
0.4
0.6
Efficiency Without bounds
With only upper bounds
CR IP T
Lace Museum 0.8
1
With lower-upper bounds
AN US
Figure 6: Comparison of the overall performance scores obtained in the second stage with lower and upper bounds on the virtual weights, with only upper bounds and without bounds.
PT
ED
M
in any perspective nor allow to concentrate the assessment of the performance on only one perspective, are really effective. We have seen in Section 5 that the constraints imposed on the virtual weights of the DEA models may regard all the museums, so that the projections of the inefficient museums on the efficient frontier satisfy all the constraints, or they may be set only with regard to the target museum. In the latter case the museum projection on the efficient frontier may not fulfil the virtual weight constraints, but this kind of constraints may allow to impose stricter virtual weights restrictions, related to sharper value judgements. Table 9 shows the performance scores obtained by imposing the virtual weight constraints only with regard to the target museum with the following upper and lower bounds for the constraints:
CE
Customer model Internal process model Innov. and learning m. Financial model Second stage model
C C C C C C C C C LC 1 = 0.6, L2 = L3 = L4 = L5 = 0; U1 = 0.8, U2 = U3 = U4 = U5 = 0.1; IP IP IP IP L1 = 0.1, L2 = 0.6; U1 = 0.4, U2 = 0.9; LI1 = 0.3, LI2 = 0.2; U1I = 0.8, U2I = 0.7; F F F F F LF 1 = 0.4, L2 = L3 = 0.05; U1 = 0.8, U2 = 0.4, U3 = 0.2; 2 2 2 2 2 2 2 L1 = 0.2, L2 = L3 = L4 = 0.1; U1 = 0.8, U2 = U3 = U42 = 0.5.
AC
Notice that these bounds are in general stricter than the bounds displayed in Table 6; indeed, these new bounds are consistent with the reasons that lead to the bounds of table 6 but they further shrink the range of the values of the virtual weights. By comparing the results displayed in Table 7 to those reported in Table 9, we may observe that the latter table presents a higher dispersion of the performance scores. Actually, several museums in the latter case exhibit a value of the performance score very close to 0. With regard to the overall performance score of the second stage model, this concerns the museums that get a very low performance score for one or more perspectives in the first stage. This is mainly due to the higher level of the 33
ACCEPTED MANUSCRIPT
Table 9: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions imposed only to the target museum.
Customer persp.
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1.0000 0.9221 1.0000 0.5980 0.5945 0.5831 0.1970 0.9674 1.0000 1.0000 0.4093
Internal process persp. 1.0000 0.4413 0.4367 0.3392 0.2081 0.1233 0.0004 0.1630 1.0000 0.0004 0.1736
Innovation and learning persp. 0.9561 1.0000 0.9153 0.9404 0.9712 0.8600 0.6906 0.9445 0.7145 0.8007 0.7915
Financial persp. 1.0000 0.2564 0.3077 0.1259 0.0879 0.1090 0.0405 0.0002 1.0000 0.1470 0.0001
1.0000 0.7042 0.7361 0.4763 0.3823 0.3573 0.0039 0.0020 0.9673 0.0042 0.0013
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
PT
7. Conclusions
ED
M
lower bounds imposed on the virtual weights in the DEA model, that do not allow to neglect the result obtained in any perspective. Therefore, in this case the DEA-BSC model with virtual weights restrictions turns out to be very strict with the museums exhibiting a very severe inefficiency as to a BSC perspective. We have also computed the performance scores for the two-stage DEA-BSC model under the assumption of constant returns-to-scale; the results are displayed in Table 10. Figure 7 compares the overall performance scores obtained with these assumption to the results of the VRS model presented in Table 7. The comparison shows that the differences are generally slight, with the exception of the Clock Tower for which the returns-to-scale seem far from being constant.
AC
CE
In this contribution we propose a two-stage model to evaluate the performance of a set of museums by combining two methodologies for the performance assessment taken from different fields. The Balanced Scorecard provides a managerial framework and is widely used in management control, while the Data Envelopment Analysis is a widely used operational research tool that allows to define a computational model. Their combination strives to get “the best of both worlds”. To our knowledge, this is the first model that exploits an integrated DEA-BSC approach for museum organisations or, more in general, for the cultural sector. The structure of the model, clearly divided into two stages, can be understood easily even by the museum managers, that do not usually have a strong background on operational research methods. This feature facilitates the interactions of the control unit with the other managers of the organisation, who can realise the aims of the methodology and how it works. In the first stage we evaluate the performance of the museums separately for each perspective suggested in the BSC framework, and this is computed with four DEA 34
ACCEPTED MANUSCRIPT
Table 10: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions for the model with CRS.
Customer persp. 1.0000 0.7910 0.9819 0.5371 0.5731 0.5737 0.3501 1.0000 0.8442 1.0000 0.5365
Innovation and learning persp. 0.9472 1.0000 0.9062 0.9305 0.9642 0.8600 0.6775 0.9348 0.7145 0.7939 0.7860
Financial persp. 1.0000 0.1697 0.3348 0.1081 0.0683 0.0668 0.0306 0.0422 0.2232 0.1001 0.1181
1.0000 0.4410 0.6528 0.3412 0.3138 0.2994 0.1876 0.3962 0.6368 0.4520 0.3058
ED
M
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
Internal process persp. 1.0000 0.3115 0.7787 0.2593 0.1593 0.1624 0.0442 0.1629 1.0000 0.3591 0.1659
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
AC
CE
PT
Doge’s Palace Correr Museum Glass Museum Ca' Rezzonico Ca' Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
0.0
0.2
0.4
0.6
0.8
1.0
Efficiency CRS
VRS
Figure 7: Comparison of the overall performance scores of stage 2 for the CRS and VRS models.
35
ACCEPTED MANUSCRIPT
CR IP T
models. In the second stage we synthesise the performance scores obtained at the first stage in a single overall performance measure. In all these DEA models we set lower and upper limits on the virtual outputs by applying the methodology that imposes proportional restrictions on virtual inputs and outputs. It is worth pointing out that the DEA models employed simultaneously adopt a variable returns-to-scale approach and set proportional restrictions on virtual outputs; moreover, these restrictions are applied both to all the museums to be evaluated and to the target museum alone. The applicability of the model proposed has been shown through an empirical investigation that has involved the municipal museums of Venice, managed by the MUVE Foundation. The results of the empirical analysis can shed light on the best practices, which are indicated for each dimension of the performance measurement process. References
AN US
[1] Allen R, Athanassopoulos A, Dyson RG, Thanassoulis E. Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions. Annals of Operations Research 1997; 73:13–34. [2] Amado CAF, Santos SP, Marques PM. Integrating the Data Envelopment Analysis and the Balanced Scorecard approaches for enhanced performance assessment. Omega 2012; 40(3):390–403.
ED
M
[3] Angulo-Meza L, Lins MPE. Review of methods for increasing discrimination in Data Envelopment Analysis. Annals of Operations Research 2002; 116(1): 225-242.
PT
[4] Aryanezhad MB, Najafi E, Farkousch SB. A BSC-DEA approach to measure the relative efficiency of service industry: A case study of banking sector. International Journal of Industrial Engeneering Computations 2011; 2(2):273–282.
CE
[5] Banker RD, Chang H, Janakiraman SN, Konstans CA. Balanced Scorecard analysis of performance metrics. European Journal of Operational Research 2004; 154(2):423–436.
AC
[6] Basso A, Funari S. Measuring the performance of museums: classical and FDH DEA models. Rendiconti per gli Studi Economici Quantitativi 2003; 2003:1–16. [7] Basso A, Funari S. A quantitative approach to evaluate the relative efficiency of museums. Journal of Cultural Economics 2004; 28(3):195–216. [8] Boorsma M, Chiaravalloti F. Arts marketing performance: An artistic-missionled approach to evaluation. The Journal of Arts Management, Law, and Society 2010; 40(4):297–317. [9] Carnegie GD, Wolnizer PW. Enabling accountability in museums. Accounting, Auditing & Accountability Journal 1996; 9(5):84–99. 36
ACCEPTED MANUSCRIPT
[10] Carvalho P, Silva Costa J, Carvalho A. The economic performance of Portuguese museums. Urban Public Economics Review 2014; 20:12–37. [11] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. European Journal of Operational Research 1978; 2(6):429–444.
CR IP T
[12] Chen TY, Chen L. DEA performance evaluation based on BSC indicators incorporated: The case of semiconductor industry. International Journal of Productivity and Performance Management 2007; 56(4):335–357. [13] Chen TY, Chen CB, Peng SY. Firm operation performance analysis using Data Envelopment Analysis and Balanced Scorecard: A case study of a credit cooperative bank. International Journal of Productivity and Performance Management 2008; 57(7):523–539.
AN US
[14] Cherchye L, Moesen W, Rogge N, Van Puyenbroeck T. An introduction to benefit of the doubt composite indicators. Social Indicators Research 2007; 82(1):111145. [15] Chiang CY, Lin B. An integration of Balanced Scorecards and Data Envelopment Analysis for firm’s benchmarking management. Total Quality Management 2009; 20(11):1153–1172.
M
[16] Cooper WW, Seiford LM, Tone K. Data Envelopment Analysis: A comprehensive text with models, applications, references and DEA-Solver Software. Boston: Kluwer Academic Publishers; 2000.
ED
[17] Del Barrio MJ, Herrero LC, Sanz JA. Measuring the efficiency of heritage institutions: A case study of a regional system of museums in Spain. Journal of Cultural Heritage 2009; 10(2):258–268.
PT
[18] Del Barrio MJ, Herrero LC. Evaluating the efficiency of museums using multiple outputs: Evidence from a regional system of museums in Spain. International Journal of Cultural Policy 2014; 20(2):221–238.
CE
[19] Eilat H, Golany B, Shtub A. R&D project evaluation: An integrated DEA and Balanced Scorecard approach. Omega 2008; 36(5):895–912.
AC
[20] Ernst D, Esche C, Erbsl¨oh U. The art museum as lab to re-calibrate values towards sustainable development. Journal of Cleaner Production 2016; 135:1446– 1460. [21] Garc´ıa-Valderrama T, Mulero-Mendigorri E, Revuelta-Bordoy D. Relating the perspectives of the Balanced Scorecard for R&D by means of DEA. European Journal of Operational Research 2009; 196(3):1177-1189. [22] Haldma T, L¨aa¨ts K. The Balanced Scorecard as a performance management tool for museums. In: Gregoriou GN, Finch N, editors. Best practices in management accounting. London: Palgrave Macmillan; 2011. p. 232–252. 37
ACCEPTED MANUSCRIPT
[23] Kaplan RS. Strategic performance measurement and management in nonprofit organizations. Nonprofit management & Leadership 2001; 11(3):353–370. [24] Kaplan RS. The Balanced Scorecard and nonprofit organizations. Balanced Scorecard Report, Balanced Scorecard Collaborative, Harvard Business School Publ 2002.
CR IP T
[25] Kaplan RS, Campbell D. Boston Lyric Opera. Cambridge: Harvard Business School; 2001. [26] Kaplan RS, Norton DP. The Balanced Scorecard–measures that drive performance. Harvard Business Review 1992; 70(1):71–79. [27] Kaplan RS, Norton DP. Transforming the Balanced Scorecard from performance measurement to strategic management Part I. Accounting Horizons 2001; 15(1):87–104.
AN US
[28] Karagiannis G, Lovell CAK. Productivity measurement in radial DEA models with a single constant input. European Journal of Operational Research 2016; 251(1):323–328. [29] Liu JS, Lu LYY, Lu W-M, Lin BJY. A survey of DEA applications. Omega 2013; 41(5):893–902.
M
[30] Lovell CAK and Pastor JT. Radial DEA models without inputs or without outputs. European Journal of Operational Research 1999; 118(1):46–51.
ED
[31] Mairesse F, Vanden Eeckaut P. Museum assessment and FDH technology: Towards a global approach. Journal of Cultural Economics 2002; 26(4):261–286.
PT
[32] Marcon G. La gestione del museo in un’ottica strategica: L’approccio della Balanced Scorecard. In: Sibilio Parri B, editor. Misurare e comunicare i risultati L’accountability del museo. Milano: Franco Angeli; 2004. p. 21–56.
CE
[33] Nøerreklit H. The balance on the balanced scorecardard – a critical analysis of some of its assumptions. Management Accounting Research 2000; 11:65–88.
AC
[34] Niven PR. Balanced Scorecard for government and nonprofit agencies. Hoboken, NJ: John Wiley; 2003. [35] Paulus O. Measuring museum performance: A study of museums in France and United States. International Journal of Arts Management 2003; 6(1):50–62. [36] PIC Green’s Sustainability Standards in Museums. A Call to Action from the Summit on Sustainability Standards in Museums 2013. Report of American Alliance of Museums. Published online http://www.aam-us.org/docs/defaultsource/professional-networks/picgreenwhitepaperfinal.pdf?sfvrsn=4:1–57.
38
ACCEPTED MANUSCRIPT
[37] Pignataro G. Measuring the efficiency of museums: A case study in Sicily. In: Rizzo I, Towse R, editors. The economics of heritage: A study in the political economy of culture in Sicily. USA: Edward Elgar Publishing Ltd; 2002. p. 65–78. [38] Podinovski VV. Production trade-offs and weight restrictions in data envelopment analysis. Journal of the Operational Research Society 2004; 5:1311-1322.
CR IP T
[39] Rickards R. Setting benchmarks and evaluating Balanced Scorecards with Data Envelopment Analysis. Benchmarking: an International Journal 2003; 10(3):226–245. [40] Rogge N. Granting teachers the “benefit of the doubt” in performance evaluations. International Journal of Educational Management 2011; 25(6):590–614.
AN US
[41] Rouse P, Putterill M, Ryan D. Integrated performance measurement design: Insights from an application in aircraft maintenance. Management Accounting Research 2002; 13(2):229–248. [42] Sarrico CS, Dyson RG. Restricting virtual weights in data envelopment analysis. European Journal of Operational Research Society 2004; 159(1):17–34.
M
[43] Shafiee M, Lofti FH, Saleh H. Supply chain performance evaluation with Data Envelopment Analysis and Balanced Scorecard approach. Applied Mathematical Modelling 2014; 38(21-22):5092–5112.
ED
[44] Taheri H, Ansari S. Measuring the relative efficiency of cultural-historical museums in Tehran: DEA approach. Journal of Cultural Heritage 2013; 14(5):431– 438.
PT
[45] Speckbacher G. The economics of performance management in nonprofit organizations. Nonnprfit Management & Leadership 2003; 13(3):267–281.
CE
[46] Thanassoulis E, Portela MCS, Allen R. Incorporating value judgements in DEA. In: Cooper WW, Seiford LW, Zhu J, editors. Handbook on Data Envelopment Analysis. Boston: Kluwer Academic; 2004.
AC
[47] Wei TL, Davey H, Coy D. A disclosure index to measure the quality of annual reporting by museums in New Zealand and the UK. Journal of Applied Accounting Research 2008; 9(1):29–51. [48] Weinstein L, Bukovinsky D. Use of the Balanced Scorecard and performance metrics to achieve operational and strategic alignment in arts and culture notfor-profits. International Journal of Arts Management 2009; 11(2):42–55. [49] Wong YHB, Beasley JE. Restricting weight flexibility in Data Envelopment Analysis. Journal of the Operational Research Society 1990; 41(9):829–835. [50] Zorloni A. Designing a strategic framework to assess museum activities. International Journal of Arts Management 2012; 14(2):31–47. 39
How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums Antonella Basso, Francesco Casarin, Stefania Funari PII: DOI: Reference:
S0305-0483(16)30580-1 10.1016/j.omega.2017.09.010 OME 1836
To appear in:
Omega
Received date: Revised date: Accepted date:
7 September 2016 23 August 2017 30 September 2017
Please cite this article as: Antonella Basso, Francesco Casarin, Stefania Funari, How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums, Omega (2017), doi: 10.1016/j.omega.2017.09.010
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • A two-stage model with weights restrictions measures the performance of museums • This is the first model that exploits an integrated DEA-BSC approach for museums
CR IP T
• In the first stage the museums performance for each BSC perspective is evaluated • In the second stage the overall performance score is computed
AC
CE
PT
ED
M
AN US
• An empirical analysis to Venice MUVE museums shows the applicability of the model
1
ACCEPTED MANUSCRIPT
How well is the museum performing? A joint use of DEA and BSC to measure the performance of museums Antonella Bassoa , Francesco Casarinb , Stefania Funaric a
CR IP T
Department of Economics, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected] b Department of Management, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected] c Department of Management, Ca’ Foscari University of Venice, Cannaregio 873, 30121 Venice, Italy, Email [email protected]
Abstract
CE
PT
ED
M
AN US
Measuring the performance has become an important issue also in the cultural sector. In the recent literature, in order to evaluate the performance of museums, Data Envelopment Analysis (DEA) models have been proposed; these models allow to take into account the multidimensional nature of the museum performance. In this paper we exploit the capability of DEA to assess the performance by integrating the DEA approach with the Balanced Scorecard (BSC) tool; this is the first study that proposes a joint application of DEA and BSC to museums. To this aim we propose a new two-stage DEA-BSC approach. In the first stage we set a Balanced Scorecard scheme devised for museums, then we define a proper DEA model for each BSC perspective and compute the DEA efficiency score for every perspective. In the second stage we define a further DEA model which combines the efficiency scores of the various BSC perspectives into an overall performance indicator. In order to prevent the computation of the efficiency score to be almost exclusively determined by one output while disregarding the others, we set proportional restrictions on the virtual outputs. Finally, the DEA-BSC joint approach is applied to the municipal museums of Venice, carrying out a detailed empirical analysis.
AC
Keywords: Performance evaluation, Data Envelopment Analysis, Balanced Scorecard, museums, virtual weights restrictions 1. Introduction This contribution intends to evaluate the performance of museums through the development of an analytical model based on the joint use of two measuring tools: Balanced Scorecard (BSC) and Data Envelopment Analysis (DEA). The model proposed may be used to allocate public or private resources to the most efficient museums and decide the amount of resources to allocate to each of them. The evaluation of the efficiency of each museum allows, in addition, the Preprint submitted to Omega
October 6, 2017
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
identification of efficient benchmarks and the dissemination of the best practices in the museum network of reference. The production of museum services is facing multiple challenges, some external and other internal to the organisation. The main external challenges for the museums management arise from the economic crisis; specifically, we observe both a reduction in the public funding for arts and a change in the consumption of culture due to a decrease in the personal income of many agents. Moreover, museums sponsors have changed their attitude towards funding and ask for a more precise and proactive assessment of the actual returns that can be achieved; even donors request a more detailed planning and information on supported activities. As a result, museums are increasingly coping with competitive situations, vying to attract both users and donors, competing with many organisations operating in the field of the free time market. From an internal point of view, the challenge for museums primarily concerns the pursuit of a clearly defined mission, focused on their social purposes. This mission must be translated into targets that are consistent not only with the social effectiveness but also with the managerial efficiency. Museums, as well as other cultural organisations (Niven [34]), may respond to the external and internal challenges by implementing suitable tools of performance measurement. Performance measurement systems aim at providing synthetic information about the efficiency and effectiveness of services and at enabling comparative assessments which are transparent, simplified and standardized. However, though performance is a multidimensional concept, the traditional systems of performance measurement tend to focus mainly on the economic-financial side and rely on balancesheet indicators to analyse the corporate performance. It has been critically remarked that the economic-financial side alone neglects to take into account other resources that are fundamental for business development, such as the skills of the staff, the trust relationship with customers and the culture of innovation (Kaplan and Norton [27]). In museums, as in the other nonprofit organisations, these problems are even worse, because of the higher level of transparency required by the presence of public funding and donors. Indeed, the (often prevailing) public funding of these organisations requires a strong need for control tools that are able to highlight and possibly measure the social impact of the activities. The combination of the assessment problems arising from the pursuit of the institutional and social purposes and the typical problems of traditional control systems (Carnegie and Wolnizer [9]) makes the control of the performance in museums particularly complex. The design of a tool for the performance measurement of museums that overcomes the drawbacks of the traditional control systems and takes into account also the social aspects is still an open issue, although there are a few contributions that discuss it (Paulus [35], Marcon [32]). In this paper we propose a quantitative model that provides a tool that measures the museums performance with a balanced approach which is able to take into consideration the efficiency of different aspects of the museum activities. In this model we combine two methodologies taken from different fields, BSC (Kaplan and Norton [27] and Kaplan [24]) and DEA (Charnes et al. [11]), 3
ACCEPTED MANUSCRIPT
CE
PT
ED
M
AN US
CR IP T
with the aim to compare the performance of a set of museums. While there is a substantial literature on BSC and DEA, and there is a number of studies that integrate the two approaches (for a review see Amado et al. [2]), there are few studies on the application of BSC to museums (see Table 1 in Section 2) and on the application of DEA to museums (see Table 2 in Section 3); on the other hand, to our knowledge this is the first study that proposes a joint application of DEA and BSC to museums. The model is organized in two stages. In the first stage we compute a DEA efficiency score for each BSC perspective, applying a Balanced Scorecard scheme specifically devised for museums. In the second stage we define a synthetic DEA model which combines the efficiency scores of the various BSC perspectives into a comprehensive performance indicator. In order to prevent the computation of the efficiency score of a museum to be almost exclusively determined by one output variable while disregarding the presence of other important outputs, we set proportional restrictions on the virtual outputs. In our experience, such an approach can be welcomed by the museum management and may be useful to take into consideration some value judgements in the evaluation process. In order to take into account the sustainability efforts undertaken by museums, we also propose a sustainability indicator for museum institutions. In the final part of the paper the models proposed are applied to a real case study in order to evaluate the performance of the museums managed by the Venice Municipal Museums Foundation (MUVE, Fondazione Musei Civici di Venezia). The empirical analysis shows that the models proposed can be effectively implemented to compare the performance of a set of museums. The structure of the paper is as follows. Sections 2 and 3 discuss the literature on the use of the Balanced Scorecard and DEA approaches for museums management and briefly present a variable returns-to-scale (VRS) DEA model for museums. The new two-stage DEA-BSC model is proposed in Section 4, where a sustainability indicator for museums is also suggested. The proportional restrictions on the virtual outputs are discussed and included in the DEA models in Section 5. Section 6 presents the data and the results of the empirical analysis. Finally, Section 7 summarises some conclusions.
AC
2. Balanced Scorecard for museum management The Balance Scorecard is a framework that links the performance of an organisation to its strategy and emphasises the connections between strategy, actions and results (Kaplan and Norton [26]). According to its original formulation, BSC has been proposed for the evaluation of profit-oriented organisations. The BSC model aims to: create a system of quantitative information on critical variables of success; counterbalance the economicfinancial data with a more complete analysis of the environment, both inside and outside the organisation, considering shareholders and customers; overcome barriers – related to the vision, the people, the ways of resources allocation, the commitment 4
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
of the management – that prevent from aligning the implementation of the strategy to its objectives. In general, BSC focuses on four perspectives according to which the organisation is evaluated: the Financial, the Customer, the Internal process, the Innovation and learning perspectives. The original formulation provides for each perspective the institutional goals and the specific objectives, the performance indicators, the target levels of the indicators and the actions to be taken in order to achieve the targets chosen. The indicators of the economic-financial (or simply “Financial”) perspective reflect the profitability (ROI, cash flow, operating income, etc.) and the ability to meet the needs of the shareholders. The Customer perspective refers to the way in which the company should be perceived by customers to be able to realise its vision; examples of indicators for this perspective are loyalty, customer satisfaction, the number of new customers, etc. From the Internal process point of view, the company explicits what must be done internally in an excellent manner to meet the expectations of shareholders and customers. Examples of indicators are the quality of the booking services and the information systems, the costs of the service production, the ability to differentiate the product. Finally, the Innovation and learning perspective can be interpreted as the ability to develop continuous improvement, innovation and learning in order to deal successfully on the medium and long term. This perspective relates to the investments aimed at increasing the capacity of systems, processes and human resources. Investments in training, equipment, the ability to increase empowerment, the staff satisfaction are examples of useful indicators. In the original framework and in many applications in the BSC literature, the basic mechanism of BSC relies on the assumption that the four perspectives are connected together by cause-effect relations, forming a system. The Innovation and learning perspective lies at the bottom of the system, and influences the Internal process perspective (quality and timing of processes); this in turn affects the Customer perspective, which in turn leads to the final Financial perspective, positioned at the top of the chain. At the centre of the system there is the strategy deriving from the vision of the management. The application of the Balanced Scorecard to nonprofit organisations, and therefore also to museums, however, faces a basic problem, which causes BSC to require some adjustments with respect to the original setting. The basic problem posed by nonprofit organisations to BSC concerns the fulfilment of the social purpose of their activity: the social purpose is usually directed toward multiple primary stakeholders groups, not only toward shareholders. Categories bearers of institutional interests, for example different groups of citizens, the local community, the public administration, even donors, appear to be stakeholders to whom the nonprofit organisation may address its actions. Moreover, the social purpose entails two further difficulties: to identify the types of primary stakeholders (see Kaplan and Norton [27]) and to understand how the social purpose is reflected in the cause-and-effect relationships between the perspec5
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
tives. Adopting an incomplete-contract view, Speckbacher [45] argues that “there is no clearly defined stakeholder group with homogeneous expectations and objectives that can be placed at the top of the hierarchy”. Speckbacher [45] also points out that while in the standard BSC the cause-and-effect chains are unidirectional, in nonprofits a multidirectional structure of cycles is typical. For nonprofit organisations, therefore, the original BSC hierarchy is challenged, and may even be arguable, given the multiple nature of their stakeholders (Norreklit [33]). Some contributions in the literature have tried to take into account these issues. According to Kaplan and Norton [27] the original BSC framework has to be adapted in order to be applied to nonprofit organisations. They suggest to place an overarching objective, representing the mission of the nonprofit organisation itself, at the top of the framework. The long-term mission should orient the BSC objectives while the Financial perspective is no longer placed at the top of the hierarchy since the financial success is not the primary objective of these organisations. Along these lines, the Financial perspective is moved from the top to the bottom line (Kaplan [23]), so that the initial financial boost is transmitted along the different perspectives until the final perspective Customer. Kaplan and Campbell [25] describe the application of this framework to the Boston Lyric Opera. Weinstein and Bukovinsky [48] apply the same scheme in a further study on the Boston Lyric Opera and underline the key role played by the employees and by the commitment of the lower hierarchical levels of the organisation in the collection of information required by BSC. On the other hand, assuming a relational and postmodern point of view on arts, Boorsma and Chiaravalloti [8] emphasize the role of the mission and of the stakeholders and develop a new version of the BSC framework, suitable for the artistic-mission led organisations. Instead of following the indication of Kaplan and Norton to consider the mission as a over-arching long-term objective and to move the Financial perspective to the bottom line of the chain, they place the mission and the stakeholder relationships on the top of the cause-effect chain and explicitly add the mission to the BSC framework as a further performance dimension. Hence, their BSC model is based on 5 dimensions; the Innovation and learning perspective, instead, is still kept at the bottom line of the chain. Their study is an interesting theoretical contribution that, however, has not been actually implemented. Indeed, the implementation of this model would require the definition of suitable indicators that measure the “different sorts of artistic value created by the arts organisation for the key stakeholder groups” and this may often be difficult. The above mentioned applications show that the scheme proposed by Kaplan and Norton [27] and Kaplan [23] may be conveniently applied to some nonprofit organisations. As for the models proposed for museum organisations, we may find a few contributions in the literature by Marcon [32], Haldma and L¨aa¨ts [22] and Zorloni [50]; with a different aim, namely to propose a disclosure index, we cite also Wei et al. [47]. These papers present different implementations within the BSC framework for the assessment of the performance of museums; for convenience, Table 1 summarises the main features of the BSC models they propose. We may observe that all the three models discussed in Marcon [32], Haldma and L¨aa¨ts [22], Zorloni [50] present a large list of detailed indicators to measure the 6
ACCEPTED MANUSCRIPT
Table 1: Literature on BSC applications to the measurement of the performance of museums.
Approach used
Context of application Perspectives: Stakeholders’
Financial Internal process
Employees and capability of the organisation
Investigations to directly check the degree of satisfaction of the public authorities and private individuals, Income from public subsidies and donations over the total revenues and income, The sum of the resources acquired and invested by the museum x a multiplication factor, Funds attracted from areas outside the region of the museum. Income (tickets, catalogs sold and other material, other services), Number of visitors, Number of complaints, Satisfaction scores obtained through surveys, Assessment of the behavior of the visitors through qualitative methods, Contingency valuations, Indicators of the value attributed to the services by customers/users (transport costs, revenues from access fees, subscriptions to educational services). Analysis of the temporal evolution of the relationship between proceeds/income and operating costs, Temporal evolution of the income to operating costs ratio, Temporal evolution of the ratio between income and expenditure. Number of restorations carried out, Number of publications, Number of items catalogued, Number of loans, Number of exhibitions, Number of guided tours, Average cost per event. Incidence of the cost of training activities on the total operating costs, Number of employees adequately trained, Number of training days per year, Ad hoc surveys, Turnover ratio. Wei, Davey and Coy [47] To examine the reporting practice of leading museums in New Zealand and the UK and to develop a museums’ performance accountability disclosure index. This paper presents a disclosure index that uses a framework based on BSC. The BSC provides a link between internal reporting of key performance issues and the need of a community organisation to report information to the public. Museums, New Zealand and UK
CE
PT
Context of application Perspectives: Institutional and social
ED
Approach used
Customer
Number of items borrowed and photocopies, Number of visiting researchers and readers, Number of study and training courses based on the collections, Number of items deposited, Number of depositing institutions, Number of consultations, Number of publications presenting the collections, Number of lectures presenting collections. Variance in operating costs, Rate variance of department budget as % of budget, Net revenue variance, Fulfilment of cost budget items. Number of units received by collections in comparison to the previous year, Number of correspondents, Number of contributors, Number of senders of images, Number of images and photos, Number of items repaired in the preservation laboratory, Number of controlled ventilated and cleaned items in the collection, Number of archived and preserved items, Number of researched museum items, Percentage of items preserved in the contemporary requirements, Number of units associated with the database, Number of items in the museum. Rate of satisfaction, Number of training courses, Rate of employees with professional certification/qualified/PHD degree, Percentage of employees aware of the museum’s vision and goals, Number of interdepartmental cooperation groups/projects, Percentage of employees supporting managerial decisions. Marcon [32] To analyse theoretically the advantages and the difficulties emerging from the adaptation of BSC to museums. After an analysis of the problems affecting existing BSC models, in the light of the challenges museums are facing, the contribution proposes a framework to adapt BSC to museums. Museums as nonprofit organisations.
M
Objective of the study
CR IP T
Objective of the study
Models description and indicators Haldma and L¨ a¨ ats [22] To examine the impact of the design and implementation of BSC on the performance measurement and management of museums After the presentation of a theoretical framework, covering the BSC implementation issues in public sector organisations and performance measurement of museums, an empirical case study is presented and discussed. Central museum, Estonia
AN US
Main features
Financial
AC
Internal process
Learning and development
Objective of the study Approach used
Context of application
7
Approach used
Context of application Perspectives: Intellectual
Percentage of works displayed, Number of artworks loaned to other museums, Number and quality of institutions to which the museum has loaned artworks, Number of artworks purchased in the previous year, Percentage of permanent collections accessioned and catalogued, Number of scholarly articles published by museum staff, Number of catalogues published, Number of curators with adjunct appointments at universities, Number of talks by curators at scholarly conferences, Number of scholarly symposiums organized by the museum, Number of research grants awarded, Number of positive critical reviews, Weekly media coverage of an exhibition, Percentage of total exhibitions presented that were organized by the museum. Range and variety of programs offered, Increase in audiences cultural knowledge and awareness, Increase in audiences knowledge of a particular author or art movement, Percentage of visitors willing to return, Increase in first-time visitors, Number of schoolchildren visiting the museum, Number of attendees at lectures, Number of artworks shown on the museums web site, Percentage of customers satisfied with programs and services, Number of people participating in educational programs versus all other available options, Number of international curators, Number of exhibitions assembled by the museum slated to travel to other international art museums, Number of collaborative projects with other cultural institutions. Availability of sensitive documents and other materials on the museums web site, Ability to meet fundraising targets, Ability to meet performance budgets, Percentage of the budget dedicated to fundraising, Fundraising efficiency (fundraising expenses divided by total contributions received), Contributed revenue as a percentage of operating revenue, In-kind sponsorships and level of earned revenue. Percentage of goals met from the most recent strategic plan or percentage of employees whose performance plans link to objectives of the strategic plan, Number of full-time curators, Number of fulltime curators with a PhD in art history, Percentage of the budget dedicated to training and professional development, Use of 360-degree feedback to evaluate staff performance, Percentage of retained employees, Percentage of employees who are satisfied with working at the museum (on a survey) and percentage of employees who are satisfied with the quality of internal communication.
and
Fi-
PT
Governance nancial
ED
M
Public
Zorloni [50] To build a model for evaluating the performance of art museums based on BSC methodology. In a first stage, the work presents a theoretical framework for assessing value creation in the museum sector and discusses the use of the BSC framework as a means for framing and focusing the strategic goals and activities of museums. In a second stage, a working set of Critical Success Factors for visual art museums is developed and used to build a framework to assess the performance. Art museums, USA and UK
AN US
Objective of the study
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
Learning and Growth
8
ACCEPTED MANUSCRIPT
AN US
CR IP T
performance with regard to the different perspectives. These indicators are very differentiated among the three models; moreover, some of them require the collection of detailed data, often obtainable only with specific analysis involving surveys or internal and reserved documents. Only Haldma and L¨aa¨ts [22] presents an application to a museum (a central museum in Estonia), while Marcon [32] and Zorloni [50] merely list the indicators they propose. Notice that Marcon [32] adds to the four classical perspectives a fifth one, the Institutional and social perspective, which is placed side by side with the Customer perspective. On the other hand, Haldma and L¨aa¨ts [22] focus the attention on stakeholders, which include customers next to other interested parties (items borrowed and photocopies, visiting researchers and readers, . . . , see Table 1), and centre the classical Innovation and learning perspective on employees (Employees and capability of the organisation perspective). As for Zorloni [50], her definition of the four BSC perspectives somewhat strays from the original BSC model: the Intellectual perspective takes the place of the Innovation and learning perspective, the Public perspective replaces the Customer one, the Governance and Financial perspective replaces the Financial one and the Learning and Growth perspective takes the place of the Internal process one. 3. DEA models to evaluate the performance of museums
AC
CE
PT
ED
M
DEA is a methodology that can be used to measure and compare the performance of homogeneous decision-making units (DMUs). DMUs refer to a group of companies, divisions, departments and administrative units that have the same goals and objectives. DEA analyses the comparative efficiency of DMUs based upon inputs and outputs. From a technical point of view, DEA allows to compute an efficiency score by solving a linear programming problem for every DMU to identify the nonparametric production frontier; this score represents a radial measure of efficiency computed with respect to the estimated efficient frontier (see e.g. Cooper et al. [16]). Originally proposed by Charnes et al. [11], DEA has a wide range of applications, from the performance evaluation of nonprofit institutions such as schools, universities, hospitals, public agencies, to the performance evaluation of banks, insurance companies, and mutual funds (for a quick review see e.g. Liu et al. [29]). Since the DEA methodology provides an overall measure of performance that is simultaneously open to multiple input and multiple output situations, the approaches based on DEA, when applied to the cultural organisations, are able to take into account the multidimensional nature of museum’s performance. Therefore, in the last decade, a growing number of researchers have adopted a DEA approach to assess the relative performance of museums; the main contributions are displayed in Table 2. Among the first contributions we may cite Mairesse and Vanden Eeckaut [31], Pignataro [37], Basso and Funari [6] and [7], while more recent contributions are Del Barrio et al. [17], Taheri and Ansari [44], Carvalho et al. [10], Del Barrio and Herrero [18]. When applied to cultural organisations, the approaches based on DEA allow to:
9
ACCEPTED MANUSCRIPT
Table 2: Literature on DEA applications to the measurement of the performance of museums. Model used
Input and output variables
CCR (CharnesCooper-Rhodes), input oriented BCC (BankerCharnes-Cooper), FDH (free disposal hull)
Basso and Funari [7]
A set of Italian municipal museums
CCR, input oriented BCC, cross efficiency
Carvalho, Silva Costa and Carvalho [10] Del Barrio, Herrero and Sanz [17]
A set of Portuguese museums
BCC
A regional system of Spanish museums
Del Barrio and Herrero [18]
A regional system of Spanish museums
Principal components and cluster analysis, CCR, input oriented BCC and superefficiency CCR, input oriented BCC, Malmquist index
Inputs: number of workers, exhibition area. Outputs: number of visitors paying the full entrance fee, number of visitors paying a reduced fee or admitted free, number of temporary exhibitions, number of other activities carried out. Inputs: number of workers, exhibition area. Outputs: number of visitors paying the full entrance fee, number of visitors paying a reduced fee or admitted free, number of temporary exhibitions, number of other activities carried out. Inputs: number of collaborators, index of facilities, number of days open to the public. Output: number of visitors. Inputs: staff, size and equipment. Output: number of visitors.
Mairesse and Vanden Eeckaut [31]
Museums from the French speaking region of Belgium
Pignataro [37]
A set of Sicilian museums
FDH-CRS (constant returns-to-scale), FDHNIRS (non increasing returns-to-scale), FDH-NDRS (non decreasing returnsto-scale), FDH-RRS (restrictive concept of returns-to-scale) CCR, BCC, Malmquist index
Taheri Ansari [44]
A set of cultural and historical museums in Tehran
CCR, MAJ (Mehrabian, Alirezaee, Jahanshahloo) full ranking method
PT
and
ED
M
Basso and Funari [6]
CR IP T
Context of application A set of Italian municipal museums
AN US
Study
Inputs: employment, size and equipment. Outputs: number of visitors, number of temporary exhibitions, a measure of social impact, percentage of artworks loaned and new acquisitions. Inputs: scientific and technical personnel, operational budget, security services. Outputs: percentage of the collection inventoried, a technical indicator, number of temporary exhibitions, number of publications, number of communication actions, opening hours, number of visitors.
Inputs: administrative and technical staff, custodians, square meters of display space. Output: number of visitors. Inputs: space and accessibility index, human resource index, facility index, introduction index. Outputs: visitors index.
AC
CE
1. take into account the multidimensional nature of museum’s performance, since the methodology provides an overall measure of performance that simultaneously considers a multiple input - multiple output situation; 2. identify a “best practice” set of museums, named peer group, which acts as a benchmark for a museum that has been evaluated as inefficient; 3. allow to identify possible paths towards efficiency.
Let us consider a set {1, 2, ..., n} of museums whose performances have to be evaluated; let us assume that each museum uses m inputs (resources required for its activities) to produce t outputs (services provided by the museum). Let us denote by Eo the efficiency score associated to a given museum o (o ∈ {1, 2, ..., n}), defined as the ratio between the weighted sum of outputs and the weighted sum of inputs. Eo can be obtained, under the assumption of variable returns-to-scale (VRS) and 10
ACCEPTED MANUSCRIPT
an output orientation, by solving the following linear programming problem in dual form, addressed as BCC (Banker-Charnes-Cooper) model: max zo +
t X
εs+ r
+
r=1
yro zo − n X
j=1 n X
n X
εs− i
(1)
i=1
yrj λj + s+ r = 0 (r = 1, ..., t)
j=1
xij λj + s− i = xio
(i = 1, ..., m)
CR IP T
subject to
m X
(2)
λj = 1
j=1
(j = 1, ..., n) (r = 1, ..., t) (i = 1, ..., m)
AN US
λj ≥ 0 s+ r ≥ 0 s− i ≥ 0
AC
CE
PT
ED
M
where yrj and xij represent the amount of output r (r = 1, . . . , t) produced and the − amount of input i (i = 1, . . . , m) used by museum j, respectively, and zo , s+ r , si and λj are the variables of the optimisation problem. For a detailed introduction to DEA models see e.g. Cooper et al. [16]. The efficiency score Eo is the reciprocal of the optimal value of zo and is comprised between 0 and 1. The value 1 characterises the efficient museums, while the lower the efficiency score, the more inefficient is the museum. For a non efficient museum (with Eo < 1) the value of the variables λj allows you to identify a set of efficient museums that constitutes a “best practice” benchmark. Such a museum could improve its efficiency by simultaneously increasing the value of its outputs while using the same P amount of inputs, or even less. By omitting the constraint nj=1 λj = 1 in the set of constraints (2) and using the same objective function (1), we obtain a model which is characterised by constant returns-to-scale (CRS), named CCR (Charnes-Cooper-Rhodes). The reciprocal of the optimal value of zo in the CRS model gives the CRS efficiency score ECRS,o and can be used to compute a measure of scale efficiency for the museums analysed, given by the ratio between the CRS and VRS efficiency score: SE o =
ECRS,o Eo
(3)
It follows that the global technical efficiency of a museum o (measured by the CRS efficiency score ECRS,o ) is given by the product of the pure technical efficiency score under VRS, Eo , times the scale efficiency factor SE o , that is: ECRS,o = Eo · SE o (see Cooper et al. [16], Basso and Funari [7]). As for the joint implementation of the DEA and BSC approaches, there are a number of studies on profit-oriented organisations, while we are not aware of applications to cultural or more in general nonprofit organisations (with the exception of some applications to the healthcare sector). 11
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
The first seminal attempts to develop models capable of exploiting complementarities between BSC and DEA date back to Rouse et al. [41] and Rickards [39], who published applications to the aircraft maintenance and construction materials, respectively. Other contributions have been published in recent years; for example some papers emphasize the potential of a joint DEA-BSC approach in the measurement of efficiency, proposing the use of a single DEA model with multiple output related to the four traditional perspectives of BSC: Chen and Chen [12] in semiconductor industry, Eilat et al. [19] to evaluate R&D projects in different stages of their lifecycle, Chen et al. [13] for a credit cooperative bank, Chiang and Lin [15] for auto companies and commercial banks. Other papers, instead, choose to work with multiple models and split the comprehensive framework in separated models associated to the various perspectives, even though in different ways: Banker et al. [5] in the US telecommunications industry, Garc´ıa-Valderrama et al. [21] in chemical-pharmaceutical R&D activities, Aryanezhad et al. [4] for a private bank, Amado et al. [2] for the department of equipment maintenance of the Portuguese delegations of a multinational company, Shafiee et al. [43] in the food industry. For example, Banker et al. [5] consider a different performance metrics for each perspective and then use a BCC DEA model with these four performance metrics as outputs. None of these studies, however, take into consideration the museums sector; the present contribution is therefore the first attempt to build a DEA-BSC model specially designed for the evaluation of the museum performance. On the other hand, the idea to measure the performance of museums separately for each BSC perspective may be interesting for museum’s top managers, as will be discussed in Section 4, and the DEA approach facilitates the performance computation. 4. A new two-stage DEA-BSC model for museum performance evaluation
AC
CE
PT
4.1. The model structure In this paper we exploit the capability of DEA to assess the museum performance by integrating the DEA approach with a BSC tool. To this aim we propose a new two-stage DEA-BSC approach. In this approach DEA models are used in the first stage to compute the relative efficiency of museums with respect to the different BSC perspectives and in the second stage to assess the overall relative efficiency. A joint DEA-BSC approach presents some advantages with respect to the use of a pure BSC approach for performance measurement. With regard to this, the benefits of the joint approach are both internal, in relation to the methodology, and external, more operational. The internal advantages regard the synergy between BSC and DEA. On the one hand, BSC allows to reduce the informational overload by focusing the attention on the key success factors, which may suggest the main variables in the DEA model. On the other hand, DEA allows to simultaneously take into account the many input and output factors of BSC, thus permitting to deal with the problem posed by the social purposes of museums, which involves different groups of stakeholders (customers, 12
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
community, professional field; see e.g. the several variables proposed in the literature and summarised in Table 1). In addition, DEA helps to overcome a problem of BSC that arises from the need to determine operational standards (Eilat et al. [19]), by obliging to translate the qualitative concepts expressed in BSC into quantitative variables. The external benefits relate to the reflections of the integrated measurement on decision-making. In the operational management the gap between the good cause of the mission and the list of operational actions to be implemented often raises issues, since the operational actions are not always effectively linked to the mission. With respect to this, the juxtaposition of BSC and DEA allows to guide the transition from the mission to the operations by helping identifying and measuring the objectives of the organisation. Indeed, by comparing different museum organisations, DEA facilitates the identification of suitable benchmarks for the organisation and the dissemination of best practices. Top museum managers have difficulties with the application of incentive mechanisms, due to the risk to worsen the conflicts inside the organisation. The joint DEA-BSC method provides them with more objective performance measures, thus supplying a more evident rationale to support the adoption of incentives. For example, the objective to improve the museum sustainability can be perceived as rather indefinite if it is not accompanied by a measurable indicator. As a matter of fact, the cells of the BSC diagram would require a set of qualitative and quantitative methods in order to be able to measure them completely. Though the DEA model may not be considered exhaustive, it can nevertheless provide an important quantitative component of the assessment of the performance. Let us remember that, in a different sector, Banker et al. [5] proposed a BSC model that measures the performance of each of the four BSC perspectives choosing a single metrics for each perspective (for example the indicator “Return on Assets” for the Financial perspective); then these metrics are considered as the output variables of a BCC DEA model without inputs, equivalent to a CCR model with a constant input. The idea of our contribution is to measure the performance of museums for each BSC perspective by taking into consideration several key indicators instead than just the main one using a DEA model. The performance assessment for all BSC perspectives may be useful because it may help the top management identify the relative efficiency of the museum processes with respect to the different focal points represented by the perspectives. Then, in a second stage, we compute an overall performance measure by applying a BCC DEA model with the DEA performance scores of the four BSC perspectives as outputs and a constant input; for a graphical representation see Figure 1. The choice of BCC models is natural, since it allows the presence of variable returns-to-scale encompassing at the same time, as a special case, the presence of constant returns-to-scale. As for the model orientation, we assume that a museum generally try to use the resources at hand in order to optimise the fulfillment of the museum mission. Under this assumption, an output orientation seems a natural choice. 13
ACCEPTED MANUSCRIPT
Internal process perspective
Innovation and learning perspective
Overall performa nce measure
Financial perspective
CR IP T
Customer perspective
Figure 1: Basic flow chart of the two-stage DEA-BSC model. The four squared boxes represent the DEA models defined in the first stage for the BSC perspectives; the round box depicts the DEA model of the second stage, which exploits the four performance scores obtained in the first stage in order to compute a unique overall performance measure.
AC
CE
PT
ED
M
AN US
Notice that we do not implement a direct hierarchy among the DEA models that measure the performance along the four BSC perspectives. This allows the model to be quite simple and easily understandable by the museum’s top managers. On the other hand, we have remarked in Section 2 that in the cultural sector the hierarchy among the BSC perspectives is not clearly defined, so much so that various scholars interpret the hierarchy in different ways or do not impose a hierarchy at all. At the same time, the separate measurement of the performance with respect to the four BSC perspectives is valued by the museum’s top managers since it may help to highlight the strengths and weaknesses of the organisation. Ultimately, we propose a two-stage DEA-BSC model organised as follows: in the first stage we identify the variables relevant for museum organisations and analyse the performance along each perspective suggested by BSC with a different DEA model, while the second stage synthesises the performance indicators of the various perspectives in an overall performance measure computed with an additional DEA model. As for the choice of the variables of the DEA models associated to the four BSC perspectives, we may observe from Table 1 that the papers on BSC for museums (Haldma and L¨a¨ats [22], Marcon [32], Zorloni [50]) list a number of different indicators; some of them, however, can only be obtained by carrying out ad hoc surveys or require the access to detailed internal data, and actually only Haldma and L¨a¨ats [22] presents an empirical application to a museum. Since our aim is to compare the performance of a set of museums, we need to choose indicators that are at the same time focused on the main characteristics of the museum activities and computable for all the museums analysed. The input (I) and output (O) variables chosen for each model are the following (see also Figure 2). Customer perspective I: insured value; it is used as a proxy for the value of the museum exhibits 14
ACCEPTED MANUSCRIPT
Input:
Outputs:
Insured value
1. Visitors 2. Web site visits 3. Members 4. Donations 5. Catalogues
CR IP T
CUSTOMER PERSPECTIVE MODEL
Input:
Outputs:
Total costs
1. Conservation, restauration costs 2. Amount spent for new acqusitions 3. Visitors
AN US
INTERNAL PROCESS PERSPECTIVE MODEL
SECOND STAGE MODEL
Outputs:
Input:
1. Customer perspective score
Constant
2. Internal process perspective score 3. Innovation and learning perspective score
Outputs:
Constant
1. Personnel training 2. Sustainability indicators: 2a. Innovative lighting 2b: Environmental sustainability 2c: Facilities for people with disabilities
PT
ED
Input:
4. Financial perspective score
M
INNOVATION AND LEARNING PERSPECTIVE MODEL
FINANCIAL PERSPECTIVE MODEL Input:
Outputs:
1. Income from tickets 2.Sponsorships, donations, public funding 3. Other incomes
AC
CE
Expenditure
Figure 2: Flow chart of the two-stage DEA-BSC model. The four boxes on the left side illustrate the input and output variables of the DEA-BSC models of the first stage; the box on the right displays the input and output variables of the DEA model of the second stage.
15
ACCEPTED MANUSCRIPT
O: number of visitors, web site visits, members, catalogues, (value of) donations Notice that the output variables in the Customer perspective relate to the stakeholders of the museum. Internal process perspective I: total costs
CR IP T
O: conservation and restoration costs, amount spent for new acquisitions, number of visitors
Innovation and learning perspective I: constant
AN US
In the Internal process perspective the output variables are related to the main purposes of the museum, namely the preservation of cultural heritage, its enhancement through an expansion of museum’s collections, and the increase in value for the museum’s customers.
O: personnel training (cost or number of hours per employee), an indicator of innovations aiming at improving the enjoyment of exhibits, an environmental indicator and an indicator of museum accessibility
CE
PT
ED
M
The outputs in the Innovation and learning perspective measure the efforts undertaken to continuously increase the skills of the museum personnel, on the one hand, and three different facets of innovation in a museum context. A more detailed discussion on how to define the innovation indicators is presented in Subsection 4.2, but they will be defined as a measure of the degree of innovation and sustainability of the museum and will not depend on the museum size. By analogy, the personnel training indicator is computed as either the cost or the number of hours per employee, hence regardless of the museum size. For the Innovation and learning perspective model, therefore, it is not necessary to consider an input variable; this situation corresponds to the case of a model with a single constant input. Financial perspective
AC
I: expenditure
O: income from tickets, sponsorships donations and public funding, other incomes
In the Financial perspective the outputs are given by museum’s income, divided into its main items. The results of the first stage models can be effectively summarised by means of an interesting geometric representation of the performance scores obtained for the four perspectives: a radar chart in which we represent along each of the four semiaxes the score of a different perspective; an example is shown in Figure 3. What we obtain is a diamond; the biggest diamond is related to the case of a museum which 16
ACCEPTED MANUSCRIPT
Customer 1.0 0.8 0.6 0.4 0.2 0.0
Innovation
Int.process
CR IP T
Financial
AN US
Figure 3: Radar chart showing the score of a different perspective along each of the four semiaxes; the values of the performance scores shown are equal to 1 for the Customer perspective, 0.5 for the Internal process perspective, 0.8 for the Innovation and learning perspective, 0.2 for the Financial perspective.
is fully efficient with respect to all the four perspectives, i.e. with efficiency scores all equal to 1, while the diamond “shrinks” asymmetrically when the efficiency score of a perspective decreases.
AC
CE
PT
ED
M
The DEA-BSC models of the first stage can be appreciated by museum’s management bodies for the detailed information provided about the performance of BSC perspectives. On the other hand, it may be useful, and certainly it arouses interest, to compute a single measure of the (relative) overall performance of the organisation. Think, for example, of the degree mark that synthesises the marks obtained in all the courses of an academic degree program: it may not be considered as essential, but it is commonly provided as a useful information. Hence, in the second stage we define a further DEA model which combines the efficiency scores obtained for the various BSC perspectives into an overall performance indicator. As is well known, the DEA approach provides a relative measure of performance, so at least one of the museums will turn out to be efficient and will receive an overall score of 1. This does not necessarily require that the efficient museums are efficient also with respect to all the four DEA-BSC models of the first stage, hence for all BSC perspectives (of course they will have to be efficient for at least some of the perspectives). Second stage model I: constant O: the efficiency scores of the DEA models for the four perspectives: Customer, Internal process, Innovation and learning and Financial perspective scores. As pointed out, two of our DEA models present a single constant input, a situation that has become less uncommon in recent years Karagiannis and Lovell [28]: both 17
ACCEPTED MANUSCRIPT
AN US
CR IP T
the Innovation and learning perspective model and the second stage model, in which the output variables are not proportional to the museum size. This feature has some interesting consequences from the point of view of the properties of the resulting DEA model. Indeed, Lovell and Pastor [30] proved that an output oriented BCC model with a single constant input, as is the case for our models, is equivalent to an output oriented BCC model without inputs, hence a “pure output” model. In addition, Lovell and Pastor [30] proved that an output oriented CCR model with a single constant input coincides with the corresponding BCC model with the same variables. Actually, an output oriented BCC model with a single constant input is equivalent to the corresponding PnCCR model, since in the case of a single constant input the convexity constraint j=1 λj = 1 is redundant. As a consequence, in the case of a single constant input the scale efficiency is equal to 1 and the returns to scale are globally constant. Note that the concept of a constant input is analogous to that of a “dummy input” equal to one for all decision-making units adopted in Cherchye et al. [14].
ED
M
4.2. Sustainability indicators for the museum activities In recent years sustainability issues have aroused more and more attention in the museum sector too; see for example [36] and Ernst et al. [20]. The efforts made in order to make the museum more sustainable, therefore, deserve to be properly taken into account when measuring the museum performance. Nevertheless, to our knowledge, scholarly literature has not yet come to propose synthetic indicators to measure museum sustainability. In this paper, as anticipated, in order to investigate whether museums are innovative and responsible for environmental impact, and whether they pay attention to social problems, we identify three groups of items:
PT
1. use of innovative lighting, 2. environmental sustainability, 3. presence of facilities for people with disability
AC
CE
and measure them by means of three numerical indices. With regard to the innovative lighting, we focus on the LED lights and measure this aspect with a binary variable that takes either value 0 (denoting the absence of LED lighting) or 1 (denoting the presence of LED lighting). In order to measure the second and third issues, we conducted some interviews with managers of the Venice Municipal Museums Foundation; as a result, we detail aspects 2 and 3 as follows: • Environmental sustainability: 1.use of energy-saving lamps; 2.presence of high efficiency systems; 3.use of water saving devices; 4.differentiated waste collection; 5.use of eco-friendly products (in the daily cleaning services, construction and restoration works and office material); 6.thermal building insulation (either total or partial); 7.monitoring of the museum’s energy consumption; 8.procurement of power supply. 18
ACCEPTED MANUSCRIPT
• Facilities for people with disability: 1. presence of lifts; 2. presence of ramps.
AN US
CR IP T
By counting the number of items present (assigning 0 when an item is absent and 1 when it is present and then summing up), we obtain an index for environmental sustainability whose values are comprised between 0 and 8 (the overall number of green items considered). Similarly, we can define an index associated to the facilities for people with disability, with values between 0 and 2. We also define a comprehensive indicator of innovation and sustainability, called sustainability indicator for the sake of brevity and computed as the weighted sum of the three indices associated to the innovative lighting, environmental sustainability and facilities for people with disability. If all items can be considered equally important (with weights equal to 1), the comprehensive indicator is simply the sum of the values of the three indices and takes values between 0 and 11, where the maximum value is reached by the museums that fulfill all items. This comprehensive indicator will be used in the empirical analysis of the Venetian museums, since the differences in some of the innovation and sustainability items were limited for the museums analysed. 5. Restricting virtual outputs weights
AC
CE
PT
ED
M
In Subsection 4.1 we have proposed a two-stage DEA-BSC model for the evaluation of the performance of museums. As usual for standard DEA models, in the computation of the performance score this model emphasises the input and output variables according to which the museum best performs. This essential feature of DEA models, however, may sometimes lead to a lack in discriminatory power, especially when we have several variables and few organisations. Actually, it may well happen that several museums are assessed as efficient, and this may regard a considerable percentage of the organisations analysed if their number is low. Moreover, the DEA efficiency may even be achieved by disregarding some important variables in the optimization process, if their weight is negligible in the optimal solution. For instance, a museum could be stated as efficient even though the number of visitors is minimal, thus neglecting an important aspect related to its mission. A way to tackle both issues at the same time is to resort to restrictions on the weights associated to the variables of the DEA program. These weights restrictions, on the other hand, allow to incorporate some value judgments that express the decision maker’s preferences in a DEA model. In the DEA literature we find various types of weights restrictions, usually grouped into four broad categories (even if we may find in the literature also a few different approaches, see e.g Rogge [40]): • absolute weights restrictions; • assurance regions of type I, that impose restrictions on the ratio between the weights of either the inputs or the outputs; • assurance regions of type II, that impose cross restrictions on the relationship between the input and output weights; 19
ACCEPTED MANUSCRIPT
• proportional restrictions on virtual inputs and outputs.
ur 0 y r 0 j ≤ Ur0 Lr0 ≤ Pt u y r rj r=1
(r0 = 1, . . . , t;
CR IP T
For a review see for example Allen et al. [1], Angulo-Meza and Lins [3] and Thanassoulis et al. [46]. With regard to our two-stage DEA-BSC model for the evaluation of the performance of museums we have decided to adopt the fourth approach, setting restrictions on virtual outputs. Proportional restrictions on virtual outputs set lower and upper limits on the proportion of each virtual output (defined as the product of the level of output r0 and the related optimal weight) with respect to the total virtual output: j = 1, . . . , n)
(4)
(i0 = 1, . . . , m;
M
vi0 xi0 j ≤ Ui0 L i 0 ≤ Pm i=1 vi xij
AN US
where Lr0 and Ur0 represent the lower and upper percentage values set for the virtual output r0 ; for example, we may require that the proportion of the total virtual output for museum j due to output r0 (say the number of visitors) is comprised between 40 % and 90 %. Virtual weights restrictions were originally proposed by Wong and Beasley [49]; for an in-depth analysis see also Sarrico and Dyson [42]. Analogously, proportional restrictions may be defined also for virtual inputs: j = 1, . . . , n)
(5)
AC
CE
PT
ED
In our model we do not apply restrictions on the virtual inputs because the DEA models considered are all based on a single input and therefore it does not make sense to consider proportional restrictions on inputs. The virtual weights restrictions defined in constraints (4) (and (5)) have the advantage to consider the relative contribution of each output (and input) to the total virtual output (and input). As such, it may be more intuitive and thus more easily understood by decision makers (see Sarrico and Dyson [42]). This is an important advantage in the museum sector, since the museum managers do not often have an operational research background. Moreover, the virtual weights restrictions allow to take into account value judgements of decision makers in a more straightforward way, and this is especially useful in the cultural sector, where we do not have a production technology that gives rise to evident technological constraints like in Podinovski [38]. In order to present the DEA-BSC models with weights restrictions on the virtual outputs, let us adopt a two-phase approach (on the two-phase solution method see for example Cooper et al. [16]). As discussed in the literature on proportional weights restrictions (Wong and Beasley [49] and Sarrico and Dyson [42]), such restrictions may be applied in two different ways. Actually, when we formulate the optimisation problem that allows to compute the efficiency score of museum o, we may either consider restrictions (4) for all the museums to be evaluated, thus adding t × n constraints to the optimisation problem, or reduce the number of weights constraints considered and add only the t constraints related to the target museum o while disregarding the others. 20
ACCEPTED MANUSCRIPT
Let us consider the first case, which corresponds to the model chosen for our analysis. Each of the four models associated to the four perspectives in the first stage, as well as the second stage model, have a similar DEA structure and may be formalised in primal form (the so called multipliers form) as follows: min vxo − vo
t X
(7)
ur yrj − vo ≥ 0
(j = 1, . . . , n)
ur0 yr0 j ≤ Ur0 L r 0 ≤ Pt r=1 ur yrj v ≥ 0, ur ≥ 0
(r0 = 1, . . . , t;
r=1
or equivalently:
(8)
j = 1, . . . , n)
AN US
vxj −
ur yro = 1
r=1 t X
CR IP T
subject to
(6)
(r = 1, . . . , t)
min vxo − vo t X
r=1
r=1
ur (Ur0 yrj ) − ur0 yr0 j ≥ 0 v ≥ 0,
ur ≥ 0
(10) (11)
(12) (j = 1, . . . , n)
ur (Lr0 yrj ) + ur0 yr0 j ≥ 0
CE
t X
AC
r=1
ur yrj − vo ≥ 0
PT
−
t X
ur yro = 1
r=1 t X
ED
vxj −
M
subject to
(9)
(13)
(r0 = 1, . . . , t;
j = 1, . . . , n)
(14)
(r0 = 1, . . . , t;
j = 1, . . . , n)
(15)
(r = 1, . . . , t)
(16)
Notice that the term −vo in the Pnobjective function (11) is associated to the variable returns-to-scale constraint j=1 λj = 1 present in the dual of the primal program (11)–(16). In the case of a model with constant returns-to-scale, instead, this term is omitted. In order to express the linear program (11)–(16) in matrix form and write the associated dual program, let us first equivalently write constraints (14) as follows: uQLj ≥ 0 (j = 1, . . . , n) 21
(17)
ACCEPTED MANUSCRIPT
where u = (u1 , . . . , ut ) is the row vector of the output weights, 0 ∈ Rt is a null row vector and QLj is the following t × t matrix: (−L1 + 1)y1j −L2 y1j ... −Lt y1j −L1 y2j (−L2 + 1)y2j . . . −Lt y2j QLj = (18) = Dj − yj L .. . −L1 ytj −L2 ytj . . . (−Lt + 1)ytj
AN US
CR IP T
Here L = (L1 , . . . , Lt ) is the row vector of the lower bounds on the proportional virtual outputs, yj = (y1j , . . . , ytj )0 is the j-th column of the t × n matrix Y of the output variables (yj is the output vector of the j-th museum) and Dj is the diagonal matrix with the elements of yj on the main diagonal. Let us observe that constraints (14) are thus written as a set of t constraints (one for each output) for each museum j (j = 1, . . . , n). By defining the block matrix QL = QL1 QL2 . . . QLn (19) the whole set of constraints (14) can be written in compact form as follows: uQL ≥ 0
(20)
Notice that QL has dimension t × (tn) and 0 is the null row vector with tn elements. Analogously, constraints (15) can be equivalently written as follows:
M
uQUj ≥ 0 (j = 1, . . . , n)
PT
ED
where QUj is the t × t matrix: (U1 − 1)y1j U2 y1j ... Ut y1j U1 y2j (U2 − 1)y2j . . . Ut y2j QUj = .. . U1 ytj U2 ytj . . . (Ut − 1)ytj
= −Dj + yj U
(21)
(22)
AC
CE
and U = (U1 , . . . , Ut ) is the row vector of the upper bounds on the proportional virtual outputs. In compact form we may define the block matrix QU = QU1 QU2 . . . QUn (23) and write the whole set of constraints (15) as follows: uQU ≥ 0
(24)
Finally, by letting X be the m × n matrix of the input variables, xo be the o-th column of X (note that in our case m = 1 since we have only 1 input variable and xo is the input of the target unit o) and 1 = (1, . . . , 1) ∈ Rt , the primal program in (multiplier) matrix form can be written as follows: min vxo − vo 22
(25)
ACCEPTED MANUSCRIPT
subject to uyo = 1 vX − uY − vo 1 ≥ 0 uQL ≥ 0 uQU ≥ 0 v ≥ 0, u ≥ 0
(26) (27) (28) (29) (30)
subject to
AN US
uyo = 1 vX − uY − vo 1 ≥ 0 uQLo ≥ 0 uQUo ≥ 0 v ≥ 0, u ≥ 0
CR IP T
Let us observe that from the linear program (25)–(30) it is immediate to deduce the primal program for the case in which the virtual weights constraints are imposed only to the target museum: min vxo − vo (31) (32) (33) (34) (35) (36)
M
We can now write the dual of the primal program (25)–(30), i.e. the model in envelopment form: max zo (37) subject to
(38) (39) (40) (41)
max s− + 1s+
(42)
Xλ + s− = xo zo∗ yo − Y λ + QL τ L + QU τ U + s+ = 0 1λ = 1 L λ ≥ 0, τ ≥ 0, τ U ≥ 0, s− ≥ 0, s+ ≥ 0
(43) (44) (45) (46)
PT
ED
Xλ ≤ xo zo yo − Y λ + QL τ L + QU τ U ≤ 0 1λ = 1 λ ≥ 0, τ L ≥ 0, τ U ≥ 0
AC
CE
where zo is the dual variable associated to constraint (26), λ ∈ Rn is the column vector of the dual variables associated to constraints (27), τ L ∈ Rtn and τ U ∈ Rtn are the column vectors of the dual variables associated to the lower and upper virtual weights constraints (28) and (29), respectively. The optimal solution of the primal and dual programs (25)–(30) and (37)–(41), denoted by zo∗ , provides the reciprocal of the efficiency score of museum o. In phase II, by setting zo = zo∗ and solving the following linear program, the sum + of the input and output slacks s− ∈ R and s+ = (s+ 1 , . . . , st ) is maximised: subject to
23
ACCEPTED MANUSCRIPT
We can now formally define the virtual weights restriction efficiency (VWRefficiency) and the projection on the VWR efficient frontier (VWR-projection) of the inefficient museums.
CR IP T
Definition 1. Let (zo∗ , λ∗ , τ L∗ , τ U ∗ , s−∗ , s+∗ ) be an optimal solution of the Phase I dual program (37)–(41) and Phase II program (42)–(46), with o ∈ { 1, . . . , n }. Museum o is called VWR-efficient if and only if this solution satisfies zo∗ = 1, s−∗ = 0 and s+∗ = 0, i.e. if it has efficiency score equal to 1 and is zero-slack. Definition 2. Let (zo∗ , λ∗ , τ L∗ , τ U ∗ , s−∗ , s+∗ ) be an optimal solution of the Phase I dual program (37)–(41) and Phase II program (42)–(46), with o ∈ { 1, . . . , n }. The VWR-projection on the efficient frontier of an inefficient museum o, (xˆo , yˆo ), is represented by:
AN US
xˆo = xo − s−∗ = Xλ∗ yˆo = zo∗ yo + QL τ L∗ + QU τ U ∗ + s+∗ = Y λ∗
(47) (48)
PT
ED
M
Similar definitions may be formulated also for the case in which the virtual weights constraints are applied only to the target museum. Nevertheless, it is worth pointing out that in this case the peer group of an inefficient museum o, i.e. the set of museums with λ∗j > 0, may include also museums that are not efficient due to the weights constraints. Moreover, in such a case the VWR-projection may not satisfy the virtual weights constraints, since it is computed as a linear combination of museums for which such constraints are not imposed. Hence the suggestion that the model in which the virtual weights restrictions are applied to all museums simultaneously may be preferable. Actually, this is the model that will be primarily used in the empirical application that will be presented in Section 6, even if in this case the lower and upper bounds have to be carefully chosen in order to ensure that the feasible region is non empty.
CE
6. An empirical analysis on the performance of the MUVE museums with the DEA-BSC model
AC
6.1. The data In this section we present an empirical application of the two-stage DEA-BSC model proposed in a real context. In this application we evaluate the relative performance of the 11 municipal museums of Venice, whose organisation, development and promotion have been entrusted to the MUVE Foundation1 , the most important Venetian museum foundation. This allows us to compare the performance of museums with similar management policies. The data on the municipal museums of Venice have been kindly provided by the MUVE Foundation and refer to the year 2013 (see Tables 3 and 4). 1
http://www.visitmuve.it/en/home/
24
ACCEPTED MANUSCRIPT
Table 3: MUVE museums: non financial data (percentage values).
29.69 16.62 4.90 14.93 9.07 9.88 4.59 1.26 0.56 1.09 7.41 100.00
Number of members 20.22 33.71 1.12 12.13 8.31 11.24 6.97 5.17 0.00 0.00 1.12 100.00
Catalogues sold 18.44 1.78 39.25 4.34 3.65 18.24 0.20 3.16 0.00 8.88 2.07 100.00
Number of web visits 50.80 12.45 5.39 5.48 5.89 5.62 3.52 1.46 1.33 1.07 7.00 100.00
Insured value
Personnel training indicator 9.96 11.48 8.78 9.48 10.45 7.44 8.44 9.60 9.67 7.47 7.24 100.00
Sustainability
57.73 12.08 5.13 7.62 4.67 4.88 1.95 0.82 0.47 0.96 3.70 100.00
ED
M
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History Museum Mocenigo Palace Carlo Goldoni’s House Clock Tower Lace Museum Fortuny Palace Total
Number of visitors 65.64 11.73 6.61 4.95 3.03 3.11 0.41 0.86 0.40 1.29 1.96 100.00
CR IP T
Total area
AN US
Museums
Table 4: MUVE museums: financial data (percentage values).
CE
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace Total
AC
Income from tickets 79.11 7.01 4.33 3.03 1.91 1.37 0.19 0.36 0.38 0.54 1.77 100.00
PT
Museums
Public funding
Sponsorships
Donations
Other incomes
Expenditure
47.79 0.00 3.82 0.00 0.00 48.39 0.00 0.00 0.00 0.00 0.00 100.00
2.69 36.10 0.00 2.98 2.89 2.63 44.28 0.00 7.09 1.34 0.00 100.00
98.04 0.00 1.08 0.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
78.22 11.32 1.19 2.73 1.75 1.93 0.63 0.13 0.13 0.15 1.81 100.00
30.95 19.31 4.01 10.70 10.71 9.06 4.34 2.77 0.55 1.69 5.92 100.00
25
Conservation costs 8.59 26.73 1.44 23.53 14.72 0.59 0.00 2.46 18.74 0.00 3.21 100.00
Personnel costs 25.55 18.93 5.72 12.97 11.75 11.38 2.99 4.01 0.53 2.04 4.12 100.00
10.23 10.23 10.23 10.23 10.23 10.23 5.68 10.23 4.55 9.09 9.09 100.00
ACCEPTED MANUSCRIPT
Table 5: MUVE museums: sustainability indicators. Innovative lighting
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1 1 1 1 1 1 1 1 0 1 1
Environmental sustainability 6 6 6 6 6 6 4 6 4 6 5
Facilities for people with disability 2 2 2 2 2 2 0 2 0 1 2
Sustainability indicator 9 9 9 9 9 9 5 9 4 8 8
CR IP T
Museums
AC
CE
PT
ED
M
AN US
It can immediately be noted the presence of a museum which is much bigger than the others. It is the Doge’s Palace, which is located in S. Mark’s square and represents a sort of “symbol of Venice”. This is by large the biggest museum of the MUVE Foundation: it attracts the majority of visitors and web visits, draws most of the donations and exhibits the highest percentages of expenditure and income earned. On the other hand, other museums are able to capture the highest number of members (Correr Museum) and to sell the highest number of catalogues (Glass Museum). Among the output variables of the model for the Internal process perspective we find the conservation and restoration costs and the amount spent for new acquisitions. With respect to the conservation and restoration costs, Correr Museum and Ca’ Rezzonico prevail. The cost for new acquisitions in the year considered, instead, is null for all the municipal museums. We gathered also information on the three groups of issues related to the presence of innovative lighting, environmental sustainability, and facilities for people with disability; in this respect, we have computed both the three relative indicators and the sustainability indicator presented in Subsection 4.2. In the empirical analysis we will consider the comprehensive sustainability indicator given the limited differences observed for the MUVE museums. From Table 5, which shows the values of the sustainability indicators for the museums analysed, we may observe that the municipal museums of Venice on the whole pay much attention to the issues related to innovation and sustainability. The last column of Table 5 displays the value of the comprehensive sustainability indicator; it is interesting to note that a good 7 museums out of 11 reach a level of 9 (with a maximum value for the indicator equal to 11), while the Mocenigo Palace and Clock Tower have wide margins for improvement with regard to sustainability. 6.2. First and second stage performance analysis In this subsection we present the results obtained by applying the two-stage DEA-BSC model with virtual weights restrictions proposed in Sections 4 and 5. 26
ACCEPTED MANUSCRIPT
Table 6: Virtual weights restrictions applied in the analysis of the MUVE museums.
LC 1 LC 2 LC 3 LC 4 LC 5
= = = = =
LIP 1 = LIP 2 =
0.4 0 0 0 0
Upper bound Ur
0 0.05
LI1 = LI2 =
0.3 0.2
LF 1 = LF 2 = LF 3 =
0.4 0 0
L21 L22 L23 L24
0.2 0.05 0.05 0.05
M
= = = =
U1C U2C U3C U4C U5C
= = = = =
0.9 0.5 0.5 0.5 0.5
CR IP T
Lower bound Lr
AN US
Virtual weight output Stage 1 Customer model Visitors Web site visits Members Donations Catalogues Internal process model Conservation, restoration costs Visitors Innovation and learning model Personnel training Sustainability indicator Financial model Income from tickets Sponsorships, donations, public funding Other incomes Stage 2 Second stage model R Customer performance score EC R Internal process performance score EIP Innovation and learning performance score EIR Financial performance score EFR
U1IP = U2IP =
0.95 1
U1I = U2I =
0.8 0.7
U1F = U2F = U3F =
0.9 0.6 0.6
U12 U22 U32 U42
0.8 0.5 0.5 0.5
= = = =
AC
CE
PT
ED
The lower and upper bounds defining the virtual weights restrictions are reported in Table 6. The choice of these bounds has been made starting from some suggestions arisen from a number of interviews to managers of the MUVE Foundation, useful to determine the relative importance of the variables. Notice that the restrictions set in Table 6 take into account the peculiarities of the context in which the museums analysed operate. For example, in the European context, donations are not so relevant as they are in the US, and this accounts for the choice to set a lower range for the value of the weight associated to this variable. On the other hand, visitors cannot be set aside given their importance in the museum mission, so the restriction on the weight associated to the number of visitors ensures that it will obtain a high value. A similar reasoning affects also the weights restrictions in the second stage model: in this model none of the perspectives can be neglected, but the importance of the performance score of the Customer perspective is emphasised with respect to the others. Moreover, we point out that the widths of the virtual weights intervals depend also on the need to have a non empty feasible region for the optimisation problems presented in Section 5, which prevents the intervals from being too narrow. Table 7 summarises the overall results of the two-stage DEA-BSC model with virtual weights restrictions. Columns 2 to 5 of this table report for each perspective the value of the performance scores computed by using the first stage models for 27
ACCEPTED MANUSCRIPT
Table 7: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions for all museums.
Customer persp.
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1.0000 0.9064 1.0000 0.5797 0.5959 0.6136 0.3589 1.0000 1.0000 1.0000 0.5570
Internal process persp. 1.0000 0.4952 0.8426 0.5120 0.3180 0.1668 0.0476 0.1666 1.0000 0.4512 0.1736
Innovation and learning persp. 0.9472 1.0000 0.9062 0.9305 0.9642 0.8600 0.6775 0.9348 0.7145 0.7939 0.7860
Financial persp. 1.0000 0.1711 0.3695 0.1110 0.0702 0.0691 0.0334 0.0493 1.0000 0.1321 0.1256
1.0000 0.5283 0.7155 0.4278 0.3743 0.3293 0.2053 0.4276 0.9840 0.5081 0.3311
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
AC
CE
PT
ED
M
the four BSC perspectives; the last column exhibits the overall performance score obtained with the second stage model. As it can be expected, the relative performance scores obtained by a museum for the different BSC perspectives may differ. In general, we may note that the number of efficient museums is relatively high for the Customer perspective (5 out of 11) while we have only 1 or 2 efficient museums for the other perspectives. This may be due to the fact that the DEA model for the Customer perspective has the highest number of output variables. On the whole, it seems that the insertion of the virtual weight constraints in the DEA models is effective in keeping the number of museums that are assessed as efficient low. It is also interesting to note that no museum is efficient with regard to all perspectives: no one is perfect. More in detail, two museums, Doge’s Palace and Clock Tower, are efficient as to the Customer, Internal process and Financial perspectives, and four other museums are efficient only as to one perspective (the Correr Museum as to the Innovation perspective, the Glass, Carlo Goldoni’s House and the Lace Museums as to the Customer one). The remaining museums do not achieve efficiency as to any perspective. The results obtained with the four DEA-BSC models at the first stage may be conveniently displayed using a radar chart as suggested by Amado et al. [2] (see Graph 1, p. 398) and are reported in Figure 4. In addition, we have also introduced in Subsection 4.1 a geometrical representation of the performance scores obtained for the four perspectives by a museum using a radar chart with the scores along the four axes (see Figure 4). Figure 5 shows the radar charts obtained for all the MUVE museums. By comparing the values of the performance scores achieved by a museum along the different perspectives we may identify its strengths and weaknesses with respect to the best performers. On the whole, the MUVE museums show better performances 28
ACCEPTED MANUSCRIPT
DP 1.0
FP
CM
0.8 0.6
LM
GM
0.4
Customer perspective 0.2 Internal process perspective 0.0
CT
CR
Innovation perspective
CGH
CP MP
NHM
CR IP T
Financial perspective
AN US
Figure 4: Efficiency scores obtained by all museums for the different perspectives in the first stage. Legend: DP:Doge’s Palace, CM:Correr Museum, GM:Glass Museum, CR:Ca’ Rezzonico, CP:Ca’ Pesaro, NHM:Natural History Museum, MP:Mocenigo Palace, CGH:Carlo Goldoni’s House, CT:Clock Tower, LM:Lace Museum, FP:Fortuny Palace.
AC
CE
PT
ED
M
as to the Customer perspective and the Innovation and learning perspective, while several museums exhibit a low performance score as to the Financial and the Internal process perspectives. In a nutshell, the MUVE museums strengths seem to be their ability to attract visitors and to arouse interest and passion, on the one hand, and the ability to train employees and to undertake sustainability actions, on the other hand. Their weaknesses, instead, seem to concern the conservation and acquisition processes as well as the fund raising operations. Moreover, we notice that the museum that is closer to the complete efficiency at the first stage is the Doge’s Palace, by large the MUVE biggest and most visited museum; and actually this is the only museum that is efficient in the second stage model which provides the overall performance indicator. As for the results obtained for the overall performance indicator in the second stage model (last column of table 7), we can note the dispersion of the performance scores. More in detail, we may distinguish 3 sets of museums: Mocenigo Palace, Natural History Museum, Fortuny Palace and Ca’ Pesaro exhibit the lowest performance scores (comprised between 0.20 and 0.37); Carlo Goldoni’s House, Ca’ Rezzonico, Lace, Correr and Glass Museums obtain an intermediate score (in the range 0.42 to 0.71), while the performance score of the Clock Tower, near 1, sets this museum close to the most efficient one (Doge’s Palace). Therefore, the DEA-BSC model with virtual weights restrictions turns out to be rather strict with the museums exhibiting an inefficiency as to one or more BSC perspectives. It is known that the DEA approach allows to suggest to the non efficient units a benchmark, given by the so called reference set, that can be looked at for improvements. From the point of view of museums, the reference set can be considered in a 29
ACCEPTED MANUSCRIPT
Correr Museum
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Financial
Innovation
Ca' Rezzonico Customer
ED
Financial
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Natural History Museum Customer
Financial
Innovation
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Innovation
PT CE AC
Financial
Int.process
Innovation
M
Customer
Financial
Int.process
Innovation
Ca' Pesaro
1.0 0.8 0.6 0.4 0.2 0.0
AN US
Customer
Financial
Int.process
Innovation
Glass Museum 1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
CR IP T
Financial
Doge’s Palace
Mocenigo Palace
Carlo Goldoni’s House
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Financial
Int.process
1.0 0.8 0.6 0.4 0.2 0.0
Innovation
Innovation
30
Int.process
ACCEPTED MANUSCRIPT
Financial
Clock Tower
Lace Museum
Customer
Customer
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
Financial
Innovation
CR IP T
Customer
Financial
Int.process
Innovation
Fortuny Palace 1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
Int.process
AN US
Innovation
Figure 5: DEA-BSC performance scores obtained for the various museums at the first stage; each radar chart shows along each of the four semiaxes the score of a different perspective.
AC
CE
PT
ED
M
sense as a benchmark utilising the best practices, to be imitated when possible. Table 8 reports the reference set of the various museums obtained with the DEA-BSC models in the first stage for the different four perspectives. The reference set of the second stage is not reported since it consists of the only efficient museum, Doge’s Palace, for all museums. As for the Internal Process and the Financial perspectives, the reference set of all inefficient museums is composed of the same pair of efficient units (Doge’s Palace and the Clock Tower), the only efficient museums. The situation is more diversified, instead, for the Customer perspective; in this case, the reference set is still composed of two efficient museums, but the pairs are not always the same. It is interesting to note that the Glass Museum enters the reference set of six other museums, while on the contrary the Clock Tower is not present in the reference set of any other. This means that six museums can draw inspiration from the Glass Museum – in combination with another efficient museum – with respect to the marketing policies related to the Customer perspective. In the end, the outcomes of the DEA-BSC model are not limited to the detection of the efficient museums, that may be used to define proper incentive mechanisms for the museums’ managers. The computation of the inefficiency scores for the non efficient museums is especially useful from the point of view of the policy and managerial implications, since it may be used to identify the weaknesses of the museum organisations. For instance, let us consider the performance score of the Mocenigo Palace for the Internal process perspective (Table 7). This is the lowest score of all MUVE museums with respect to this perspective, and this is indicative of the fact that this museum fails to reach the goals of the Internal process. By looking more closely at the data, 31
ACCEPTED MANUSCRIPT
Table 8: Reference sets of the various museums obtained with the DEA-BSC models of the first stage for the different four perspectives. Legend: DP:Doge’s Palace, CM:Correr Museum, GM:Glass Museum, CR:Ca’ Rezzonico, CP:Ca’ Pesaro, NHM:Natural History Museum, MP:Mocenigo Palace, CGH:Carlo Goldoni’s House, CT:Clock Tower, LM:Lace Museum, FP:Fortuny Palace.
{DP} {DP, GM} {GM} {DP, GM} {GM, LM} {GM, CGH} {GM, LM} {CGH} {CT} {LM} {GM, LM}
Innovation persp.
Financial persp.
{CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM} {CM}
{DP} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {DP, CT} {CT} {DP, CT} {DP, CT}
CR IP T
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Nat. Hist. M. Mocenigo Palace C. Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
Internal process persp. {DP} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {DG, CT} {CT} {DG, CT} {DG, CT}
AN US
Customer persp.
ED
M
we may see that this museum exhibits at the same time relatively high total costs and few visitors, and in the year considered it does not fulfill the conservation and restoration objective in any way. Hence, an improvement is certainly advisable for this museum as to the Internal process perspective. Since the Mocenigo Palace is a museum of textiles and costumes, new well promoted temporary exhibitions centered on historical Venetian textiles might be able to arouse the interest of foreign tourists visiting Venice and to attract a higher number of visitors.
AC
CE
PT
6.3. Changing the model options We have seen that the results obtained for the DEA-BSC model with virtual weights restrictions indicate that the museums that are inefficient with respect to one o more BSC perspectives are rather severly penalised in the overall performance indicator of stage 2. In order to see if this penalisation is worsened by the presence of the bounds in the model, we have investigated the effect of the inclusion of the upper and lower bounds in the second stage model. More precisely, we have compared the overall performance scores obtained in the second stage considering the restrictions reported in Table 6 to the results calculated with the analogous model without any second stage restriction and with the analogous model including only the upper bounds. The comparison of the results are summarised in Figure 6. As can be see, the results obtained with and without bounds are rather different. In detail, if we compare the performance scores of the unrestricted model to those of the model with upper bounds, it is clear that the introduction of the upper bounds does affect the results. Moreover, when we include also the lower bounds, the performance scores further sensibly decreases. Therefore the bounds imposed on the virtual weights in the DEA model, that neither allow to neglect the result obtained 32
ACCEPTED MANUSCRIPT
Doge’s Palace Correr Museum Glass Museum Ca' Rezzonico Ca' Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Fortuny Palace 0
0.2
0.4
0.6
Efficiency Without bounds
With only upper bounds
CR IP T
Lace Museum 0.8
1
With lower-upper bounds
AN US
Figure 6: Comparison of the overall performance scores obtained in the second stage with lower and upper bounds on the virtual weights, with only upper bounds and without bounds.
PT
ED
M
in any perspective nor allow to concentrate the assessment of the performance on only one perspective, are really effective. We have seen in Section 5 that the constraints imposed on the virtual weights of the DEA models may regard all the museums, so that the projections of the inefficient museums on the efficient frontier satisfy all the constraints, or they may be set only with regard to the target museum. In the latter case the museum projection on the efficient frontier may not fulfil the virtual weight constraints, but this kind of constraints may allow to impose stricter virtual weights restrictions, related to sharper value judgements. Table 9 shows the performance scores obtained by imposing the virtual weight constraints only with regard to the target museum with the following upper and lower bounds for the constraints:
CE
Customer model Internal process model Innov. and learning m. Financial model Second stage model
C C C C C C C C C LC 1 = 0.6, L2 = L3 = L4 = L5 = 0; U1 = 0.8, U2 = U3 = U4 = U5 = 0.1; IP IP IP IP L1 = 0.1, L2 = 0.6; U1 = 0.4, U2 = 0.9; LI1 = 0.3, LI2 = 0.2; U1I = 0.8, U2I = 0.7; F F F F F LF 1 = 0.4, L2 = L3 = 0.05; U1 = 0.8, U2 = 0.4, U3 = 0.2; 2 2 2 2 2 2 2 L1 = 0.2, L2 = L3 = L4 = 0.1; U1 = 0.8, U2 = U3 = U42 = 0.5.
AC
Notice that these bounds are in general stricter than the bounds displayed in Table 6; indeed, these new bounds are consistent with the reasons that lead to the bounds of table 6 but they further shrink the range of the values of the virtual weights. By comparing the results displayed in Table 7 to those reported in Table 9, we may observe that the latter table presents a higher dispersion of the performance scores. Actually, several museums in the latter case exhibit a value of the performance score very close to 0. With regard to the overall performance score of the second stage model, this concerns the museums that get a very low performance score for one or more perspectives in the first stage. This is mainly due to the higher level of the 33
ACCEPTED MANUSCRIPT
Table 9: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions imposed only to the target museum.
Customer persp.
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
1.0000 0.9221 1.0000 0.5980 0.5945 0.5831 0.1970 0.9674 1.0000 1.0000 0.4093
Internal process persp. 1.0000 0.4413 0.4367 0.3392 0.2081 0.1233 0.0004 0.1630 1.0000 0.0004 0.1736
Innovation and learning persp. 0.9561 1.0000 0.9153 0.9404 0.9712 0.8600 0.6906 0.9445 0.7145 0.8007 0.7915
Financial persp. 1.0000 0.2564 0.3077 0.1259 0.0879 0.1090 0.0405 0.0002 1.0000 0.1470 0.0001
1.0000 0.7042 0.7361 0.4763 0.3823 0.3573 0.0039 0.0020 0.9673 0.0042 0.0013
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
PT
7. Conclusions
ED
M
lower bounds imposed on the virtual weights in the DEA model, that do not allow to neglect the result obtained in any perspective. Therefore, in this case the DEA-BSC model with virtual weights restrictions turns out to be very strict with the museums exhibiting a very severe inefficiency as to a BSC perspective. We have also computed the performance scores for the two-stage DEA-BSC model under the assumption of constant returns-to-scale; the results are displayed in Table 10. Figure 7 compares the overall performance scores obtained with these assumption to the results of the VRS model presented in Table 7. The comparison shows that the differences are generally slight, with the exception of the Clock Tower for which the returns-to-scale seem far from being constant.
AC
CE
In this contribution we propose a two-stage model to evaluate the performance of a set of museums by combining two methodologies for the performance assessment taken from different fields. The Balanced Scorecard provides a managerial framework and is widely used in management control, while the Data Envelopment Analysis is a widely used operational research tool that allows to define a computational model. Their combination strives to get “the best of both worlds”. To our knowledge, this is the first model that exploits an integrated DEA-BSC approach for museum organisations or, more in general, for the cultural sector. The structure of the model, clearly divided into two stages, can be understood easily even by the museum managers, that do not usually have a strong background on operational research methods. This feature facilitates the interactions of the control unit with the other managers of the organisation, who can realise the aims of the methodology and how it works. In the first stage we evaluate the performance of the museums separately for each perspective suggested in the BSC framework, and this is computed with four DEA 34
ACCEPTED MANUSCRIPT
Table 10: Results from the first and second stage of the DEA-BSC model with virtual weights restrictions for the model with CRS.
Customer persp. 1.0000 0.7910 0.9819 0.5371 0.5731 0.5737 0.3501 1.0000 0.8442 1.0000 0.5365
Innovation and learning persp. 0.9472 1.0000 0.9062 0.9305 0.9642 0.8600 0.6775 0.9348 0.7145 0.7939 0.7860
Financial persp. 1.0000 0.1697 0.3348 0.1081 0.0683 0.0668 0.0306 0.0422 0.2232 0.1001 0.1181
1.0000 0.4410 0.6528 0.3412 0.3138 0.2994 0.1876 0.3962 0.6368 0.4520 0.3058
ED
M
Doge’s Palace Correr Museum Glass Museum Ca’ Rezzonico Ca’ Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
Internal process persp. 1.0000 0.3115 0.7787 0.2593 0.1593 0.1624 0.0442 0.1629 1.0000 0.3591 0.1659
AN US
Museums
Second Stage Overall indicator
CR IP T
First Stage
AC
CE
PT
Doge’s Palace Correr Museum Glass Museum Ca' Rezzonico Ca' Pesaro Natural History M. Mocenigo Palace Carlo Goldoni’s H. Clock Tower Lace Museum Fortuny Palace
0.0
0.2
0.4
0.6
0.8
1.0
Efficiency CRS
VRS
Figure 7: Comparison of the overall performance scores of stage 2 for the CRS and VRS models.
35
ACCEPTED MANUSCRIPT
CR IP T
models. In the second stage we synthesise the performance scores obtained at the first stage in a single overall performance measure. In all these DEA models we set lower and upper limits on the virtual outputs by applying the methodology that imposes proportional restrictions on virtual inputs and outputs. It is worth pointing out that the DEA models employed simultaneously adopt a variable returns-to-scale approach and set proportional restrictions on virtual outputs; moreover, these restrictions are applied both to all the museums to be evaluated and to the target museum alone. The applicability of the model proposed has been shown through an empirical investigation that has involved the municipal museums of Venice, managed by the MUVE Foundation. The results of the empirical analysis can shed light on the best practices, which are indicated for each dimension of the performance measurement process. References
AN US
[1] Allen R, Athanassopoulos A, Dyson RG, Thanassoulis E. Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions. Annals of Operations Research 1997; 73:13–34. [2] Amado CAF, Santos SP, Marques PM. Integrating the Data Envelopment Analysis and the Balanced Scorecard approaches for enhanced performance assessment. Omega 2012; 40(3):390–403.
ED
M
[3] Angulo-Meza L, Lins MPE. Review of methods for increasing discrimination in Data Envelopment Analysis. Annals of Operations Research 2002; 116(1): 225-242.
PT
[4] Aryanezhad MB, Najafi E, Farkousch SB. A BSC-DEA approach to measure the relative efficiency of service industry: A case study of banking sector. International Journal of Industrial Engeneering Computations 2011; 2(2):273–282.
CE
[5] Banker RD, Chang H, Janakiraman SN, Konstans CA. Balanced Scorecard analysis of performance metrics. European Journal of Operational Research 2004; 154(2):423–436.
AC
[6] Basso A, Funari S. Measuring the performance of museums: classical and FDH DEA models. Rendiconti per gli Studi Economici Quantitativi 2003; 2003:1–16. [7] Basso A, Funari S. A quantitative approach to evaluate the relative efficiency of museums. Journal of Cultural Economics 2004; 28(3):195–216. [8] Boorsma M, Chiaravalloti F. Arts marketing performance: An artistic-missionled approach to evaluation. The Journal of Arts Management, Law, and Society 2010; 40(4):297–317. [9] Carnegie GD, Wolnizer PW. Enabling accountability in museums. Accounting, Auditing & Accountability Journal 1996; 9(5):84–99. 36
ACCEPTED MANUSCRIPT
[10] Carvalho P, Silva Costa J, Carvalho A. The economic performance of Portuguese museums. Urban Public Economics Review 2014; 20:12–37. [11] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. European Journal of Operational Research 1978; 2(6):429–444.
CR IP T
[12] Chen TY, Chen L. DEA performance evaluation based on BSC indicators incorporated: The case of semiconductor industry. International Journal of Productivity and Performance Management 2007; 56(4):335–357. [13] Chen TY, Chen CB, Peng SY. Firm operation performance analysis using Data Envelopment Analysis and Balanced Scorecard: A case study of a credit cooperative bank. International Journal of Productivity and Performance Management 2008; 57(7):523–539.
AN US
[14] Cherchye L, Moesen W, Rogge N, Van Puyenbroeck T. An introduction to benefit of the doubt composite indicators. Social Indicators Research 2007; 82(1):111145. [15] Chiang CY, Lin B. An integration of Balanced Scorecards and Data Envelopment Analysis for firm’s benchmarking management. Total Quality Management 2009; 20(11):1153–1172.
M
[16] Cooper WW, Seiford LM, Tone K. Data Envelopment Analysis: A comprehensive text with models, applications, references and DEA-Solver Software. Boston: Kluwer Academic Publishers; 2000.
ED
[17] Del Barrio MJ, Herrero LC, Sanz JA. Measuring the efficiency of heritage institutions: A case study of a regional system of museums in Spain. Journal of Cultural Heritage 2009; 10(2):258–268.
PT
[18] Del Barrio MJ, Herrero LC. Evaluating the efficiency of museums using multiple outputs: Evidence from a regional system of museums in Spain. International Journal of Cultural Policy 2014; 20(2):221–238.
CE
[19] Eilat H, Golany B, Shtub A. R&D project evaluation: An integrated DEA and Balanced Scorecard approach. Omega 2008; 36(5):895–912.
AC
[20] Ernst D, Esche C, Erbsl¨oh U. The art museum as lab to re-calibrate values towards sustainable development. Journal of Cleaner Production 2016; 135:1446– 1460. [21] Garc´ıa-Valderrama T, Mulero-Mendigorri E, Revuelta-Bordoy D. Relating the perspectives of the Balanced Scorecard for R&D by means of DEA. European Journal of Operational Research 2009; 196(3):1177-1189. [22] Haldma T, L¨aa¨ts K. The Balanced Scorecard as a performance management tool for museums. In: Gregoriou GN, Finch N, editors. Best practices in management accounting. London: Palgrave Macmillan; 2011. p. 232–252. 37
ACCEPTED MANUSCRIPT
[23] Kaplan RS. Strategic performance measurement and management in nonprofit organizations. Nonprofit management & Leadership 2001; 11(3):353–370. [24] Kaplan RS. The Balanced Scorecard and nonprofit organizations. Balanced Scorecard Report, Balanced Scorecard Collaborative, Harvard Business School Publ 2002.
CR IP T
[25] Kaplan RS, Campbell D. Boston Lyric Opera. Cambridge: Harvard Business School; 2001. [26] Kaplan RS, Norton DP. The Balanced Scorecard–measures that drive performance. Harvard Business Review 1992; 70(1):71–79. [27] Kaplan RS, Norton DP. Transforming the Balanced Scorecard from performance measurement to strategic management Part I. Accounting Horizons 2001; 15(1):87–104.
AN US
[28] Karagiannis G, Lovell CAK. Productivity measurement in radial DEA models with a single constant input. European Journal of Operational Research 2016; 251(1):323–328. [29] Liu JS, Lu LYY, Lu W-M, Lin BJY. A survey of DEA applications. Omega 2013; 41(5):893–902.
M
[30] Lovell CAK and Pastor JT. Radial DEA models without inputs or without outputs. European Journal of Operational Research 1999; 118(1):46–51.
ED
[31] Mairesse F, Vanden Eeckaut P. Museum assessment and FDH technology: Towards a global approach. Journal of Cultural Economics 2002; 26(4):261–286.
PT
[32] Marcon G. La gestione del museo in un’ottica strategica: L’approccio della Balanced Scorecard. In: Sibilio Parri B, editor. Misurare e comunicare i risultati L’accountability del museo. Milano: Franco Angeli; 2004. p. 21–56.
CE
[33] Nøerreklit H. The balance on the balanced scorecardard – a critical analysis of some of its assumptions. Management Accounting Research 2000; 11:65–88.
AC
[34] Niven PR. Balanced Scorecard for government and nonprofit agencies. Hoboken, NJ: John Wiley; 2003. [35] Paulus O. Measuring museum performance: A study of museums in France and United States. International Journal of Arts Management 2003; 6(1):50–62. [36] PIC Green’s Sustainability Standards in Museums. A Call to Action from the Summit on Sustainability Standards in Museums 2013. Report of American Alliance of Museums. Published online http://www.aam-us.org/docs/defaultsource/professional-networks/picgreenwhitepaperfinal.pdf?sfvrsn=4:1–57.
38
ACCEPTED MANUSCRIPT
[37] Pignataro G. Measuring the efficiency of museums: A case study in Sicily. In: Rizzo I, Towse R, editors. The economics of heritage: A study in the political economy of culture in Sicily. USA: Edward Elgar Publishing Ltd; 2002. p. 65–78. [38] Podinovski VV. Production trade-offs and weight restrictions in data envelopment analysis. Journal of the Operational Research Society 2004; 5:1311-1322.
CR IP T
[39] Rickards R. Setting benchmarks and evaluating Balanced Scorecards with Data Envelopment Analysis. Benchmarking: an International Journal 2003; 10(3):226–245. [40] Rogge N. Granting teachers the “benefit of the doubt” in performance evaluations. International Journal of Educational Management 2011; 25(6):590–614.
AN US
[41] Rouse P, Putterill M, Ryan D. Integrated performance measurement design: Insights from an application in aircraft maintenance. Management Accounting Research 2002; 13(2):229–248. [42] Sarrico CS, Dyson RG. Restricting virtual weights in data envelopment analysis. European Journal of Operational Research Society 2004; 159(1):17–34.
M
[43] Shafiee M, Lofti FH, Saleh H. Supply chain performance evaluation with Data Envelopment Analysis and Balanced Scorecard approach. Applied Mathematical Modelling 2014; 38(21-22):5092–5112.
ED
[44] Taheri H, Ansari S. Measuring the relative efficiency of cultural-historical museums in Tehran: DEA approach. Journal of Cultural Heritage 2013; 14(5):431– 438.
PT
[45] Speckbacher G. The economics of performance management in nonprofit organizations. Nonnprfit Management & Leadership 2003; 13(3):267–281.
CE
[46] Thanassoulis E, Portela MCS, Allen R. Incorporating value judgements in DEA. In: Cooper WW, Seiford LW, Zhu J, editors. Handbook on Data Envelopment Analysis. Boston: Kluwer Academic; 2004.
AC
[47] Wei TL, Davey H, Coy D. A disclosure index to measure the quality of annual reporting by museums in New Zealand and the UK. Journal of Applied Accounting Research 2008; 9(1):29–51. [48] Weinstein L, Bukovinsky D. Use of the Balanced Scorecard and performance metrics to achieve operational and strategic alignment in arts and culture notfor-profits. International Journal of Arts Management 2009; 11(2):42–55. [49] Wong YHB, Beasley JE. Restricting weight flexibility in Data Envelopment Analysis. Journal of the Operational Research Society 1990; 41(9):829–835. [50] Zorloni A. Designing a strategic framework to assess museum activities. International Journal of Arts Management 2012; 14(2):31–47. 39